You are not logged in.
http://ck-hack.blogspot.com/2010/11/cre … mment.html
I've had every man and his dog either drop into IRC or email me asking me what my thoughts are on the grouping tasks by tty layer patch discussed here: Phoronix link and slashdot. I guess people didn't understand my 2.6.36-ck1 announcement clearly enough, so I'll quote it again here:
Those following the development of the patches for interactivity at massive
load, I have COMPLETELY DROPPED them as they introduce regressions at normal
workloads, and I cannot under any circumstances approve changes to improve
behaviour at ridiculous workloads which affect regular ones. I still see
precisely zero point at optimising for absurd workloads. Proving how many
un-niced jobs you can throw at your kernel compiles is not a measure of one's
prowess. It is just a mindless test.Remember, I already had developed a hierarchical tree-based penalty patch for BFS and blogged about it here. I can do it in a 10 line patch for BFS, but it introduced regressions, which is why I dropped it (see earlier blog entry here: further-updates-on-hierarchical-tree).
Again, I can't for the life of me see why you'd optimise for make -j64 on a quad core machine. It is one workload, unique to people who compile all the time, but done in a way you wouldn't normally do it anyway. It is not going to magically make anything else better. If for some god-forsaken reason you wanted to do that, you could already do that with nice, or even better, by running it SCHED_IDLEPRIO.
nice -19 make -j 64 blahblah
or
schedtool -D -e make -j64 blahblahIt's not really that hard folks...
And if you really really really still want the feature for BFS, the patch that does the hierarchical tree based penalty is rolled into a bigger patch (so a lot more than just the 10 lines I mentioned) that can also group threads and enable/disable the features and it's still here:
bfs357-penalise_fork_depth_account_threads.patch
It is worth noting also that the mainline approach costs you in throughput, whereas this patch is virtually free.
EDIT: I forgot to mention that for YEARS now I've been using my "toolsched" wrapper scripts that do this automatically. See toolsched for the scripts. Make always starts as SCHED_IDLEPRIO for me at home.
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
Hm, how about running apps from a menu with something like
urxvt -e smplayer
yeah, a terminal (with it's own pseudo-tty) would be started along with the app, but still...
Offline
So much for the hype...
Some authors will have to do some serious explanations to their readers, like this one:
The patch by Linux kernel developer Mike Galbraith adds a mere 233 lines of code to the kernel's scheduler, but it cuts desktop latency down by a factor of ten. That's impressive — it's almost like getting a new computer.
Arch linux
Offline
Hm, how about running apps from a menu with something like
urxvt -e smplayer
yeah, a terminal (with it's own pseudo-tty) would be started along with the app, but still...
The whole idea is that desktop apps are (usually) interactive, so they will always have priority in this system. Note that this feature is called "autogroup", if you want to manually start something backgrounded or at a high priority, nice/ionice/schedtool will work a lot better than this cgroup hack.
Offline
Anything not launched from a virtual terminal or a terminal emulator won't get autogrouped into a cgroup, so this patch ignores normal desktop applications. That just means they get treated as interactive though (and the terminal apps are prevented from slowing them down).
This patch ONLY helps you if you do stuff in the terminal that used to cause desktop lag.
If I get the article posted by slytux right (http://www.linusakesson.net/programming/tty/index.php), this is a missunderstanding : tty are not only used by the console, but are a sort of interface system used for any communication between processes and user input/output. A modern system runs many tty in parallel (ls /dev/tty* ). If I understood correctly, the patch autogroup into a cgroup the processes running on top of each tty ; and it somehow improve interactivity of the processes running in other tty/cgroup.
Offline
ps -e
Will show you if a process is attached to a tty (pts == pseudo tty, for terminal emulators, and they work with this patch) - desktop applications don't have ttys, they are for terminal processes that take input/output, even daemons don't have ttys.
If you run a desktop application that spawns a bunch of threads/processes that heavily use CPU (a web browser playing 2-3 flash videos in different tabs), this patch won't improve the performance of your movie player running at the same time.
Luckily, desktop applications don't usually eat up a bunch of CPU cycles when you aren't interacting with them (unlike compiling in a terminal).
Last edited by thestinger (2010-11-17 21:56:44)
Offline
yes, but the X server process is attached to a tty...
What you see on your screen is not displayed by the application process directly, but through the X server.
Last edited by bzhb (2010-11-17 22:24:45)
Offline
If you run a desktop application that spawns a bunch of threads/processes that heavily use CPU (a web browser playing 2-3 flash videos in different tabs), this patch won't improve the performance of your movie player running at the same time.
Can you please explain why not, if we run both apps from terminal in seperate ttys?
Offline
Read the blog post by Con Kolivas that graysky posted and his response to this question:
nit said...
You might be true, but this patch still be good for desktop "normal" users, that have no idea of what tty, nice, task and kernel is. They like to open browsers with a lot of tabs, with a lot of Adobe Flash Player and other plugins, while listening to music and using some chat program.
ck said...
See that's the thing, it won't. The rest of all those normal desktop GUI apps won't be grouped. If you -do- group all firefox threads together as one app, or all amarok threads together as one app, both of those will perform -worse- under load.
Basically, when you decide to compile with a ridiculous number of jobs, instead of CFS giving each process equal resources, it will consider that all the processes are in one cgroup and won't let them monopolize 90% of the cpu time, keeping your desktop responsive.
I think people are interpreting this as a fix for the IO load problem, which is something completely different.
Last edited by thestinger (2010-11-17 22:55:50)
Offline
yes, but the X server process is attached to a tty...
What you see on your screen is not displayed by the application process directly, but through the X server.
X is attached to the tty, but not the processes running in it. Being displayed through X isn't the same as running through X. Notice that when you run firefox from a terminal, it is attached to a tty (use ps). Not so when running from Xorg. If all the applications under Xorg were in the same tty group, they would be less responsive under this patch, because CFS would consider them part of the same task.
Read the kernel mailing list that's been linked to... it explains everything.
Last edited by thestinger (2010-11-17 22:57:07)
Offline
okay, I understand the problem now
Offline
Basically, when you decide to compile with a ridiculous number of jobs, instead of CFS giving each process equal resources, it will consider that all the processes are in one cgroup and won't let them monopolize 90% of the cpu time, keeping your desktop responsive.
I think people are interpreting this as a fix for the IO load problem, which is something completely different.
Right. I still fail to understand this clearly, so correct me if I'm wrong.
Basically everyone is saying that "GUI apps don't have a tty". Well, I think they do, at least here under Openbox - it's called tty1. So, basically this patch does nothing if all apps are run on tty1. What I'm asking, is that if we ran for example a 1080p movie (which uses quite a bit resources) from some generic player that was ran from it's own terminal (and thus it's own pseudo-tty), and some resource-heavy site in firefox in another pseudo-tty, would browsing performace improve (things like scrolling, zooming etc.) with this patch, because each app is now running in it's own tty and this patch would distribute CPU resources more effectively?
Offline
Each firefox process (flash player, plugin container, firefox-bin), would receive less priority than it usually would. Same thing for the movie player.
So if firefox was running under a tty and decided to spawn 100 processes, your movie wouldn't stall like it would without the patch (assuming it was under a different tty).
If all your apps are under tty1, this patch will probably make your desktop less responsive.
Last edited by thestinger (2010-11-17 23:13:24)
Offline
Each firefox process (flash player, plugin container, firefox-bin), would receive less priority than it usually would. Same thing for the movie player.
So if firefox was running under a tty and decided to spawn 100 processes, your movie wouldn't stall like it would without the patch (assuming it was under a different tty).
If all your apps are under tty1, this patch will probably make your desktop less responsive.
Right, thats what I thought. All my apps ARE under tty1 because that's my login tty, and I run all my apps inside it (inside Openbox), and I think so do most of the users who run any kind of GUI desktop.
So basically this patch prevents processes inside one tty to go haywire and choke the CPU to death, making just enough room for other ttys to function properly. Why would I ever want this except when compiling some extra-heavy stuff, I don't know.
If all your apps are under tty1, this patch will probably make your desktop less responsive.
If this is true, this patch would make more harm than good for "normal" desktop users. However, I can't imagien why this would be true - I mean, this patch shouldn't affect processes running in the same tty.
Offline
your desktop processes shouldn't under a tty though... (did you really get that output from ps -e?)
a tty usually has one interactive process at most
thestinger@arch i ~ % ps -e
PID TTY TIME CMD
1 ? 00:00:00 init
2 ? 00:00:00 kthreadd
3 ? 00:00:00 ksoftirqd/0
4 ? 00:00:00 kworker/0:0
6 ? 00:00:00 migration/0
7 ? 00:00:00 migration/1
8 ? 00:00:00 kworker/1:0
9 ? 00:00:00 ksoftirqd/1
10 ? 00:00:08 kworker/0:1
11 ? 00:00:00 migration/2
12 ? 00:00:00 kworker/2:0
13 ? 00:00:00 ksoftirqd/2
14 ? 00:00:00 migration/3
15 ? 00:00:00 kworker/3:0
16 ? 00:00:00 ksoftirqd/3
17 ? 00:00:00 cpuset
18 ? 00:00:00 khelper
19 ? 00:00:00 netns
20 ? 00:00:00 pm
21 ? 00:00:00 sync_supers
22 ? 00:00:00 bdi-default
23 ? 00:00:00 kblockd
24 ? 00:00:00 kacpid
25 ? 00:00:00 kacpi_notify
26 ? 00:00:00 kacpi_hotplug
27 ? 00:00:00 kseriod
28 ? 00:00:00 kworker/1:1
29 ? 00:00:01 kworker/2:1
30 ? 00:00:00 kworker/3:1
31 ? 00:00:00 kswapd0
32 ? 00:00:00 ksmd
33 ? 00:00:00 fsnotify_mark
34 ? 00:00:00 aio
35 ? 00:00:00 crypto
37 ? 00:00:00 nouveau
38 ? 00:00:00 ttm_swap
39 ? 00:00:00 kstriped
40 ? 00:00:00 hd-audio0
552 ? 00:00:00 ata_sff
603 ? 00:00:00 scsi_eh_0
604 ? 00:00:00 scsi_eh_1
605 ? 00:00:00 scsi_eh_2
606 ? 00:00:00 scsi_eh_3
607 ? 00:00:00 scsi_eh_4
608 ? 00:00:00 scsi_eh_5
668 ? 00:00:00 kdmflush
671 ? 00:00:00 kdmflush
673 ? 00:00:00 kdmflush
676 ? 00:00:00 kdmflush
679 ? 00:00:00 kdmflush
695 ? 00:00:00 jbd2/dm-2-8
696 ? 00:00:00 ext4-dio-unwrit
733 ? 00:00:00 udevd
2063 ? 00:00:00 khubd
2180 ? 00:00:00 cfg80211
2223 ? 00:00:00 scsi_eh_6
2224 ? 00:00:00 usb-storage
2226 ? 00:00:00 phy0
2296 ? 00:00:00 usbhid_resumer
2339 ? 00:00:00 udevd
2341 ? 00:00:00 udevd
2413 ? 00:00:00 jbd2/dm-1-8
2414 ? 00:00:00 ext4-dio-unwrit
2421 ? 00:00:00 jbd2/dm-3-8
2422 ? 00:00:00 ext4-dio-unwrit
2435 ? 00:00:00 jbd2/dm-4-8
2436 ? 00:00:00 ext4-dio-unwrit
2684 tty2 00:00:00 mingetty
2685 tty3 00:00:00 mingetty
2686 ? 00:00:00 su
2699 ? 00:00:00 xinit
2700 ? 00:00:00 syslog-ng
2701 ? 00:00:00 syslog-ng
2711 ? 00:00:00 acpid
2722 tty4 00:18:23 X
2775 ? 00:00:00 pdnsd
2806 ? 00:00:00 crond
2811 ? 00:00:01 dropbox
2841 ? 00:00:01 irqbalance
2857 ? 00:00:00 ntpd
2870 ? 00:00:00 dbus-daemon
2895 ? 00:00:00 flush-254:3
2896 ? 00:00:00 flush-254:4
2902 ? 00:00:53 awesome
2908 ? 00:00:00 clipboard-daemo
2909 ? 00:00:00 unclutter
2912 ? 00:00:19 urxvtd
2921 ? 00:00:00 mpd
2928 ? 00:00:00 dbus-launch
2929 ? 00:00:00 dbus-daemon
3118 pts/0 00:00:00 zsh
3259 pts/0 00:00:00 mutt
3281 pts/1 00:00:01 zsh
10389 pts/2 00:00:00 zsh
12371 pts/2 00:00:00 sudo
12373 pts/2 00:00:00 zsh
15294 ? 00:00:00 kworker/u:2
18500 ? 00:03:10 firefox-bin
19023 ? 00:00:00 kworker/u:0
21571 pts/1 00:01:16 Steam.exe
21574 ? 00:00:13 wineserver
21578 ? 00:00:00 services.exe
21580 ? 00:00:00 winedevice.exe
21587 ? 00:00:00 explorer.exe
22171 pts/0 00:00:00 vimpager
22214 pts/0 00:00:00 vim
22262 ? 00:00:00 kworker/u:1
22322 ? 00:00:00 rgvim
22325 pts/1 00:00:00 ps
gvim, firefox, etc. have no pseudo-tty since they don't start from a terminal, steam does because I started it from a terminal (actually gvim was started from a terminal, but it forks)
Last edited by thestinger (2010-11-17 23:38:07)
Offline
your desktop processes shouldn't under a tty though... (did you really get that output from ps -e?)
a tty usually has one interactive process at most
My output:
PID TTY TIME CMD
1 ? 00:00:00 init
2 ? 00:00:00 kthreadd
3 ? 00:00:00 ksoftirqd/0
6 ? 00:00:00 migration/0
7 ? 00:00:00 watchdog/0
8 ? 00:00:00 cpuset
9 ? 00:00:00 khelper
10 ? 00:00:00 netns
11 ? 00:00:00 pm
12 ? 00:00:00 sync_supers
13 ? 00:00:00 bdi-default
14 ? 00:00:00 kblockd
15 ? 00:00:00 kacpid
16 ? 00:00:00 kacpi_notify
17 ? 00:00:00 kacpi_hotplug
18 ? 00:00:00 kseriod
19 ? 00:00:01 kworker/0:1
20 ? 00:00:00 khungtaskd
21 ? 00:00:00 kswapd0
22 ? 00:00:00 ksmd
23 ? 00:00:00 fsnotify_mark
24 ? 00:00:00 aio
25 ? 00:00:00 crypto
65 ? 00:00:00 nouveau
66 ? 00:00:00 ttm_swap
525 ? 00:00:00 ata_sff
586 ? 00:00:00 scsi_eh_0
587 ? 00:00:00 scsi_eh_1
589 ? 00:00:00 scsi_eh_2
607 ? 00:00:00 scsi_eh_3
609 ? 00:00:00 scsi_eh_4
625 ? 00:00:00 scsi_eh_5
626 ? 00:00:00 kworker/u:3
646 ? 00:00:00 kworker/u:5
714 ? 00:00:00 jbd2/sda2-8
715 ? 00:00:00 ext4-dio-unwrit
745 ? 00:00:00 udevd
1499 ? 00:00:00 edac-poller
1500 ? 00:00:00 khubd
1794 ? 00:00:00 kpsmoused
2121 ? 00:00:00 cfg80211
2296 ? 00:00:00 phy0
2391 ? 00:00:00 scsi_eh_6
2392 ? 00:00:00 usb-storage
2768 ? 00:00:00 usbhid_resumer
3193 ? 00:00:00 saa7133[0]
3610 ? 00:00:00 jbd2/sda4-8
3611 ? 00:00:00 ext4-dio-unwrit
3615 ? 00:00:00 udevd
3618 ? 00:00:00 mount.ntfs-3g
3621 ? 00:00:00 mount.ntfs-3g
3624 ? 00:00:00 mount.ntfs-3g
3738 ? 00:00:00 syslog-ng
3739 ? 00:00:00 syslog-ng
3768 ? 00:00:00 dbus-daemon
3771 tty1 00:00:00 login
3772 tty2 00:00:00 agetty
3773 tty3 00:00:00 agetty
3774 tty4 00:00:00 agetty
3775 tty5 00:00:00 agetty
3776 tty6 00:00:00 agetty
3777 ? 00:00:00 flush-8:0
3795 ? 00:00:00 dhcpcd
3797 tty1 00:00:00 bash
3804 tty1 00:00:00 startx
3821 tty1 00:00:00 xinit
3822 tty7 00:01:05 X
3825 tty1 00:00:00 openbox
3830 tty1 00:00:00 dbus-launch
3831 ? 00:00:00 dbus-daemon
3841 tty1 00:00:00 tint2
3843 tty1 00:00:00 clipboard-daemo
3844 tty1 00:00:02 conky
3845 tty1 00:00:00 conky
3848 tty1 00:00:07 conky
3855 tty1 00:00:08 conky
3863 tty1 00:00:01 conky
3901 tty1 00:00:53 midori
3988 ? 00:00:00 udevd
4354 ? 00:00:00 kworker/0:2
4364 tty1 00:00:00 qtfm
4386 ? 00:00:00 kworker/0:0
4399 tty1 00:00:00 urxvt
4401 pts/1 00:00:00 bash
4420 tty1 00:00:00 pidgin
4424 tty1 00:00:00 filezilla
4439 pts/1 00:00:00 ps
Running midori, qtfm, filezilla, pidgin, and the apps defined at startup... They all run inside a tty1 O.o
EDIT: I think it has to do with the way Openbox and Awesome start their apps...
Last edited by karabaja4 (2010-11-17 23:48:30)
Offline
3804 tty1 00:00:00 startx
3825 tty1 00:00:00 openbox
it's because you use startx, so all your processes are running under that virtual terminal
I start X with xinit in inittab
x:5:once:/bin/su thestinger -l -c "/usr/bin/xinit ~/.config/xinitrc -- /usr/bin/X -nolisten tcp vt04 >/dev/null 2>&1"
2902 ? 00:00:53 awesome
Most people use display managers, and their desktop apps won't be under a tty afaik.
Offline
That's fine and shouldn't matter. If you run a compile in another terminal, you don't want that slowing your box to a crawl and as far as the end-user is concerned it shouldn't matter if the compile takes a bit longer because of using a few *desktop* apps.
Last edited by slytux (2010-11-18 00:05:09)
Offline
It's not about the compile taking a bit longer - if your desktop is under tty1 because you start x from a virtual terminal, your individual *desktop* apps will end up with less cpu time than your daemons and other stuff without a tty.
Again, this doesn't apply to 90%+ of people because they use a display manager, so the patch will work fine.
Last edited by thestinger (2010-11-18 00:09:12)
Offline
You can also disable it :-)
Offline
I have a feeling it will be improved by the time 2.6.38 is ready anyway .
Offline
It's not about the compile taking a bit longer - if your desktop is under tty1 because you start x from a virtual terminal, your individual *desktop* apps will end up with less cpu time than your daemons and other stuff without a tty.
Again, this doesn't apply to 90%+ of people because they use a display manager, so the patch will work fine.
Damn, you have a point there. I guess I'll have to make my apps tty-less soon.
So basically,
without the patch - all apps, tty-less or not, get the full amount of CPU attention.
with the patch - tty-less apps get full CPU attention, and those inside ttys are given less CPU priority and thus prevented from hogging the CPU.
Did I finally get it right this time?
Last edited by karabaja4 (2010-11-18 02:48:19)
Offline
without the patch - all apps, tty-less or not, get the full amount of CPU attention.
correct
with the patch - tty-less apps get full CPU attention,
yes
those inside ttys are prevented from hogging the CPU
sort of yes:
this is how I understand it: usually processes are treated equally, but with the patch, processes in a tty are grouped together, so a tty with 1 process won't show a difference, but a tty with 60 processes won't be able to hog the cpu nearly as much
Last edited by thestinger (2010-11-18 00:31:50)
Offline
Aha, I see. thestinger, thanks alot for the clarifications, I understand alot better now.
And now I'm even more sure about one thing - this patch doesn't really improve performance or latencies for normal desktop user. All it does is stopping the processes in a single tty to rape the CPU's brains out. So phoronix and slashdot created this big hype out of this little hack...
I'm guessing this is just a proof that Linux users crave responsivness on desktop - it was discussed in a another thread that compared Linux responsivness to that of Windows systems - and now is in Topics Going Nowhere - unfairly in my opinion, because the topic deserved a discussion. I'm not happy with Linux responsivness on desktop, and always looking for ways to speed things up... But I have gone offtopic
P.S. and judging by the kernel mailing list, the same functionality of this patch can be done by editing .bashrc, and thus using the benefits of this patch with schedulers like BFS...
.
Last edited by karabaja4 (2010-11-18 02:47:18)
Offline
Well tweaking the scheduler is one of the tasks that was set for 2.6.37 anyway, before this patch was brought to public.
Offline