bad performance of the kernel scheduler, is this normal?

zasdfgbnm · 2016-12-24 08:11:43

Hi All,

I have a PC with i7-6950x CPU (10 cores, 20 threads) and 128G RAM. I installed both the official kernel and the ck kernel, they are all the newest version.

$ pacman -Q linux linux-ck-broadwell
linux 4.8.13-1
linux-ck-broadwell 4.8.15-2

I have a computation intensive application, which simple create 20 threads and run all computation in parallel. The application takes ~60GB RAM while running.

What happened on my computer is:

Under official kernel, if I run my application, the overall CPU utilization can not reach 100% and the CPU temperature is lower than 40C. If I play a music at the same time, it is so laggy that sometimes there is no sound for a short while and sometimes different piece of music get messed up...
Under the ck kernel, things looks much better: the overall CPU utilization easily keeps at 100% and the CPU temperature reaches ~70C. If I play a music at the same time, the music goes smoothly. But I can still feel laggy when I try to switch between different windows and browse webpages.

I also tested compiling linux kernel while listening to music and watching movies, and I got similar result.

This behavior is very strange to me, because I have been using Linux for ~6 years and since the SCHED_AUTOGROUP patch and before 2016, linux kernel perform excellent on its schedule: even if I open 5 times more computation intensive processes than the number of threads of the CPU, I could not feel any lag at all, the system just kept running smoothly.

But unfortunately at sometime in 2016, things changed and linux can not keep smooth any more if my CPU is busy on computation intensive programs. Same behavior is observed on all my computers. Since archlinux has been the only distro I use for a long time, I don't know whether I will observe similar behavior on other distros, but I guess this problem should not depend on distro.

Can anyone tell me that if your system is having the same behavior? Does anyone know what has changed in recent linux kernels that cause this problem?

Last edited by zasdfgbnm (2016-12-24 08:17:38)

brebs · 2016-12-24 11:29:16

zasdfgbnm wrote:

But I can still feel laggy when I try to switch between different windows and browse webpages.

Are you using the "nice" (or "renice") command, to reduce the priority of the CPU-hogging processes?

I've always found Con's schedulers to be smoother.

graysky · 2016-12-24 11:47:40

brebs wrote:

zasdfgbnm wrote:
But I can still feel laggy when I try to switch between different windows and browse webpages.
Are you using the "nice" (or "renice") command, to reduce the priority of the CPU-hogging processes?
I've always found Con's schedulers to be smoother.

+1 for this:

% grep makepkg ~/.zshrc
alias makepkg='nice -19 makepkg'

Also, with 128G of RAM you should be doing all your compiling in tmpfs if you aren't already:

% grep scratch /etc/fstab
tmpfs /scratch             tmpfs nodev,relatime,size=28G 0 0

(I only have 32 GB so I use 28 for tmpfs).

Last edited by graysky (2016-12-24 11:49:01)

zasdfgbnm · 2016-12-25 05:56:06

brebs wrote:

zasdfgbnm wrote:
But I can still feel laggy when I try to switch between different windows and browse webpages.
Are you using the "nice" (or "renice") command, to reduce the priority of the CPU-hogging processes?
I've always found Con's schedulers to be smoother.

Thanks for this trick. I know that reducing the priority will make it smoother, and I was not using this trick on the test stated in this thread.

What I'm wondering is, according to my experience, the official scheduler used to work so well (on my computer) that we don't even need to bother setting the priority manually to make it smooth. But now, we can only get the same level of smooth by manually setting the priority.

Also, another problem is, under official scheduler, as I said in #1, the overall CPU utilization can not reach 100%, which is totally a weird behavior. I don't know what the official scheduler is doing...

Last edited by zasdfgbnm (2016-12-25 05:56:45)

zasdfgbnm · 2016-12-25 06:03:36

graysky wrote:

brebs wrote:
zasdfgbnm wrote:
But I can still feel laggy when I try to switch between different windows and browse webpages.
Are you using the "nice" (or "renice") command, to reduce the priority of the CPU-hogging processes?
I've always found Con's schedulers to be smoother.
+1 for this:
% grep makepkg ~/.zshrc
alias makepkg='nice -19 makepkg'
Also, with 128G of RAM you should be doing all your compiling in tmpfs if you aren't already:
% grep scratch /etc/fstab
tmpfs /scratch             tmpfs nodev,relatime,size=28G 0 0
(I only have 32 GB so I use 28 for tmpfs).

Yes, it's a good idea to run everything in RAM if possible.

I just checked on the official document:

https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt wrote:

size: The limit of allocated bytes for this tmpfs instance. The
default is half of your physical RAM without swap. If you
oversize your tmpfs instances the machine will deadlock
since the OOM handler will not be able to free that memory.

So the default option "size=64GB" is perfect to me

Arch Linux

#1 2016-12-24 08:11:43

bad performance of the kernel scheduler, is this normal?

#2 2016-12-24 11:29:16

Re: bad performance of the kernel scheduler, is this normal?

#3 2016-12-24 11:47:40

Re: bad performance of the kernel scheduler, is this normal?

#4 2016-12-25 05:56:06

Re: bad performance of the kernel scheduler, is this normal?

#5 2016-12-25 06:03:36

Re: bad performance of the kernel scheduler, is this normal?

Board footer