New Linux Versions = Decreased performance?

[martinp]

Hey everyone, I am in the process of upgrading one of our servers and I have noticed a decrease in performance from newer versions of Linux.

To perform this test I have a large file and doing a basic read cycle test.  The file contains:

     Maximum Record size ..........: 256 (Variable)
     Maximum # records ............: (No limit)
     Current # records ............: 1867559
     Size of key block ............: 4096 bytes
     Record Expansion factor ......: 10%
     External key size ............: 0
     Alt. key 0 ...................: [1:1:6]+[2:1:8]+[3:1:2]+[4:1:3]+[5:1:5]+[6:1:2]+[7:1:16]


Read Cycle Program:

0010 BEGIN
0020 PRINT DTE(0:"%hz:%mz:%sz")
0030 OPEN (1)"/home/martin/TESTFILE"
0040 LET X$=KEY(1,END=0099)
0050 READ (1,KEY=X$)A$
0060 T++
0070 GOTO 0040
0099 PRINT DTE(0:"%hz:%mz:%sz")

The important part of this test is that it is to be run *MULTIPLE* times to fill the memory.  Usually two passes complete filling the memory.  Afterwards zero disk access is used, it's all memory based.  This eliminates the disk-io variable from the test.

Test Results:

Code Select Expand


ubuntu server 10.04 = 9 seconds 2.6.32-38-server
ubuntu server 12.04 = 9 seconds 3.2.0-23-generic
ubuntu server 16.04 = 9 seconds 4.4.0-21-generic
ubuntu server 18.04 = 14 seconds 4.15.0-128-generic #131 (running pxp for ub16)
ubuntu server 20.04 = 14 seconds 5.4.0-58-generic #64

debian 10 = 14 seconds 4.19.0-13-amd64

windows10 = 11 seconds  20H2 - 19042.685


All tests were done with pvx17 64-bit and the same hardware, however I've seen the same results with older providex as well.

So my current conclusions are something has happened around the 4.4+ kernel that seems to be causing this as you can see a 5 second difference between 16.04 and 18.04.  I'm not sure what the solution is yet, as I have just discovered this after numerous tests, I'd like to hear from the staff at pvxplus before I do any more tests.

Thanks for any thoughts,
Martin

[Mike King]

You might want to check the size of the disc cache and buffers on the various platforms.

Try:   free -m  (The -m indicates output MB)

The newer operating system may be using more memory (not uncommon) resulting in less space in memory to buffer data.

[martinp]

Thanks Mike, I checked the free -m on two of them both appear to fill up the cash properly during the test and remain as I re-test.  I see no disk access during the tests.

The reason I am reporting this, is I notice across the board performance reduction, though it may not be as 'apparent' to the average user.

ubuntu 12:

             total       used       free     shared    buffers     cached
Mem:          7890       1056       6834          0         17        883
-/+ buffers/cache:        155       7735
Swap:         8092          0       8092


ubuntu 20:

              total        used        free      shared  buff/cache   available
Mem:           7865         171        6688           2        1005        7447
Swap:          8191           0        8191

[martinp]

I also did another test by creating a ramdisk and transferred the test file to the ramdisk and the results were exactly as when the file is fully cached, no improvement.

I looked into some of the kernel tuning parameters, and I can't find anything to really get this to go any faster like the earlier kernels.

[Mike King]

About a year ago there was a issue that hit the news about a potential security leak in Intel processors.  The problem had to do with one process reading the cache of another process.

The correction, which for most was done via software, was notable in that it slowed down the systems quite significantly.  Perhaps this is what you are seeing.

Here is a link:  https://www.extremetech.com/computing/261420-early-data-shows-linux-update-fix-intel-security-flaw-hits-performance-hard

[martinp]

Bingo!  We got it Mike!

I can't believe the performance hit this has resulted in.

There are ways to disable the fixes on the linux kernel which are available by research but I won't post those there due to security.  However I am glad the reason was found.  I am personally surprised at the level of impact it has on pvx.  It's probably because pvx is so fast as well that its a full single thread process that any delays are exasperated.

I had updated my ubuntu16 test machine and afterwards it too was patched and began to have the slow down.

Even though the test example I gave was an extreme case, I did notice a noticeable performance hit across the board.  Now in order to overcome these fixes, one needs to purchase newer hardware that are not affected by these security flaws.

Thanks again for your help on this, I'm not sure if this will help others or just help answer my stubbornness to understand the issue.  At least I have some baselines and examples to compare with now.

Have a great new years