Arch Linux

kubast2

Member

Registered: 2018-09-16

Posts: 6

Balancing out latency and throughput with IO scheduler tunables/kernel

Hey, I got a dm-cache ssd+hdd / software sshd setup, and while I am very satisfied with boot times, the desktop latency was more or less not the best thing ever.

Can I switch to blk-mq after majority of userspace loaded?

Here is a thing:
Start ups, shutdowns and sddm/kde loading takes a whole lot more time when blk-mq + bfq on hdd/data device [ssd/cache device uses mq-deadline with default settings] is enabled. < default kernel; However tty's load immediately, apps react immediately to sigterms, downloading+taking a screenshot,  krunner launches after the spectacle i/o ends.
Overall bfq ends up having consistent latency(around 5-15 secs for a screenshot selection).

While on default io schedulers(cfq iirc), start ups, system shutdowns and kde loading after sddm login are pretty fast, however the overall latency of using the desktop is of the roof when something isn't ssd cached, apps don't react pretty much to sigterm's, tty's take 2 minutes to load+login, often times even though the screen looks locked launching krunner from shortcut and typing in "killall appname" works a lot faster than root sigterm from a tty even if I remove the fact I need to turn on tty etc.
Overall cfq ends up having variable latency of rather huge range(from ssd/application-cached-in-ram level to if you don't shutdown that app it will take at least 10 minutes to take a selection screenshot).

I am using an udev rule to set bfq

ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/iosched/slice_idle/us}="500"

I also used default slice_idle setting overall setting it to 500 was the most acceptable setting for taking a screenshot when downloading stuff. Both default and 500 are far from a decent experience when booting up.

My questions:
1. Are -zen -ck -bfq-mq -liquirix etc. kernels a placebo effect or they are fine tuned?
2. How can I tune cfq/bfq for more consistent-latency/better boot times?
3. What are other good options for hdd io schedulers?

If all else fails:
I make a backup
I setup ssd for / + /home/$USER/.cache
Or setup ssd for / and hdd for ~/Videos ~/Pictures ~/Downloads ~/Desktop ~/Documents ~/.local/share/Steam

brebs

Member

Registered: 2007-04-03

Posts: 3,742

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

tty's take 2 minutes to load+login

Investigate that problem. It's more than just the scheduler. Is the system (CPU, drives) being completely overloaded? Any interesting log/dmesg messages?

kubast2

Member

Registered: 2018-09-16

Posts: 6

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

tty's take 2 minutes to load+login

Investigate that problem. It's more than just the scheduler. Is the system (CPU, drives) being completely overloaded? Any interesting log/dmesg messages?

Yesterdays dmesg looks good.
Checking in yesterdays journalctl right now, checking "warning" logging level.
Besides a hexchat core dump/crash that I don't recall happening(from my perspective I just closed the hexchat window).
a ton of them are re-occuring every couple of minutes/hours and/or happen in batches of 3-10 messages in one minute, besides the hexchat core dump there aren't a lot of non-reoccuring messages(I skipped cpu throttling due to temperature since it throttles to 2.6-2.8Ghz, there was one message "CPU1 CPU2 CPU3 shouldn't be asleep").
The reoccurring ones:

wrz 19 21:49:12 archlinux plasmashell[1607]: file:///usr/share/plasma/plasmoids/org.kde.plasma.mediacontroller/contents/ui/main.qml:63:38: Unable to assign [undefined] to bool
wrz 19 21:51:10 archlinux plasmashell[1607]: file:///usr/share/plasma/plasmoids/org.kde.desktopcontainment/contents/ui/FolderItemDelegate.qml:300:17: QML QQuickText: Binding loop detected for property "width"
wrz 19 22:00:23 archlinux krunner[633]: QSqlDatabasePrivate::addDatabase: duplicate connection name '/home/kubast2/.mozilla/firefox/hu8yu16r.default/favicons.sqlite', old connection removed.
wrz 19 22:08:19 archlinux kwin_x11[629]: qt.qpa.xcb: QXcbConnection: XCB error: 9 (BadDrawable), sequence: 60197, resource id: 41943848, major code: 14 (GetGeometry), minor code: 0
wrz 19 23:55:59 archlinux plasmashell[1607]: UdevQt: unhandled device action "bind"
wrz 19 23:55:59 archlinux kdeinit5[576]: UdevQt: unhandled device action "bind"
wrz 19 23:55:59 archlinux baloo_file[631]: UdevQt: unhandled device action "bind"
wrz 19 23:55:59 archlinux org_kde_powerdevil[679]: UdevQt: unhandled device action "bind"
wrz 20 00:16:39 archlinux baloo_file_extractor[17177]: avformat_open_input error:  -1094995529 < times 10
wrz 20 00:00:56 archlinux plasmashell[1607]: qt.qpa.xcb: QXcbConnection: XCB error: 2 (BadValue), sequence: 12740, resource id: 67109290, major code: 142 (Unknown), minor code: 3
wrz 20 00:24:42 archlinux kwin_x11[629]: qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 23286, resource id: 37756204, major code: 18 (ChangeProperty), minor code: 0
wrz 20 00:24:42 archlinux kwin_x11[629]: qt.qpa.xcb: QXcbConnection: XCB error: 8 (BadMatch), sequence: 23446, resource id: 29360143, major code: 42 (SetInputFocus), minor code: 0

The last 3 happen every time kwin mildly freezes or some window is unusable during i/o load, like for example mouse focus doesn't get forwarded(happens on both) or else.

Overall it happens when x.org/kwin_x11 completely freezes and it isn't possible to even move windows around.
It is an laptop i5-7200U cpu.
The only thing that runs in terms of random read/writes are usually log writtes, lvm cache syncing, firefox cache, and maybe at times kde's indexer. It is very likely that besides cpu wait time cpu gets overloaded.
Steam chat is now a web browser so it definitely uses more cpu, I also have a web browser running with a youtube video(240p for audio only or automatic-1080p-) some random articles and websites open and qbittorrent making a constant 2.5-3.4 MiB/s load on hdd.

Cpu load:
On bfq the cpu load is 25-50% across all cores with steam chat+firefox+qbittorrent+pulseeffects occasionally 3-4 cores hit 100% cpu usage when you count in cpu wait time. And I got konsole+htop open right now. But no freezes of xorg/kwin happen. Albeit I usually batch download torrents so it is around 12 of them running at the same time[the main difference is in number of seeds and peers so I can see the cpu usage without cpu wait time to be much higher, number of files is the same] . qbittorrent freezes still happen, but it doesn't take the entire system down with it and it somewhat is recoverable without having to drop to tty or having to shutdown an application.
Ram load: 2GB/7.66GB 4GB used with cache;

Last edited by kubast2 (2018-09-20 10:42:03)

brebs

Member

Registered: 2007-04-03

Posts: 3,742

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

cpu throttling due to temperature

That is a problem to be fixed, not ignored. High temperatures can cause problems/instability with other components, e.g. motherboard, drives, RAM. Check their temperatures, using e.g. lm_sensors.

kubast2

Member

Registered: 2018-09-16

Posts: 6

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

cpu throttling due to temperature

That is a problem to be fixed, not ignored. High temperatures can cause problems/instability with other components, e.g. motherboard, drives, RAM. Check their temperatures, using e.g. lm_sensors.

It only happens when gaming or during a very long ffmpeg encode, the fan is running at 100% with both custom and stock fan curve.
I have already replaced thermal paste with high performance thermal grizzly compound(which increased cpu frequency by 200 mhz), tightened all the cooler screws.
I also use cooler master's usb fan cooling pad, the only way I could fix those thermals would be by either running a max frequency at 2600/2500 mhz under stock frequency.
[sarcasm]
The only thing left for me is:
Putting the laptop in a freezer.
Creating my own cooler/using a desktop one, one of which is very uneconomical and the other removes convenience of a laptop.
[/sarcasm]
The temps of the hdd/ssd are allright.  Cpu of course under stress is on fire at 92-97*C, the only device that always has a very high temperature is the pcie wifi card, but I have no issues with it, even though sensors report that ath10k_hwmon-pci-0300 (atheros 10k 03:00.0 Network controller: Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter (rev 31)
) has 82*C even on completely cold boot.
So I put on a gaming load:
I can check the temp of pch under load, but chipset is built into cpu so it most likely will hit a high temperature(well the pch has a delta of around -20*C away from the cpu temperature).
HDD temp: 37*C
SSD temp: 42*C
Wifi card: 82*C regardless of load and how far away from boot time it is.
PCH/Mobo: 20*C difference from a cpu(under load) or cpu temps(on idle)
ACPI thermal zone: 91*C
Cpu: 92-95 *C
Cpu frequency jumps around: 2950-3100 mhz might be because the summer is over and the air temperature went down.

No idea about the temps on laptops, but the last laptops that have never throttled on me where from core 2 duo era.
So when it comes to temperature I am unable to do anything, and given that I have replaced the thermal paste/had to unscrew the cooler and screw it back in, they won't accept a out of charge warranty service on this laptop(I presume they won't).
RAM
HDD
ssd( is a bit closer to cpu )
Are on the other side of this laptop motherboard and are futher away from cpu ,gpu and it's cooler.

Last edited by kubast2 (2018-09-20 13:59:20)

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 78,840

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

When you talk about 5-15 seconds or 10 minutes for "screenshots", do you *literally* mean "screenshots" (like, make an image of the things on your monitor) or is that a reference to eg. filesystem snapshots?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

kubast2

Member

Registered: 2018-09-16

Posts: 6

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

When you talk about 5-15 seconds or 10 minutes for "screenshots", do you *literally* mean "screenshots" (like, make an image of the things on your monitor) or is that a reference to eg. filesystem snapshots?

Short: I mean it literally.

Long:
Fullscreen screenshots take as fast as it takes to launch Spectacle(kde literal screenshots tools), which is quiet fast,
What I exactly mean by screenshots here I literally mean turning on Spectacle, clicking on create new screenshot from selection, selecting the area that I want to screenshot, and then confirm that this is the area of the screen I want to screenshot.
So it isn't just taking a screenshot of entire screen, because that usually takes 0.01s up to 20 secs it would take to launch spectacle.
On cfq with no load it takes no more than 0.5-2 secs to do that, but under i/o load it can take anywhere from 1 minute to 10 minutes/when it takes over a minute I need to force kill spectacle and try again.
On bfq seems to take under minute/half-minute, but the minimal time is 7 secs regardless of load it takes at least 7 secs.

^All of those are measured like this:
0.Close previous spectacle instance.
0.note: Continued custom selection screenshots on bfq and cfq under no load without closing spectacle take less and less time, but I don't leave the screenshot app open at all times.
1.Launch spectacle.
2.Create new screenshot from custom selection.
3.I confirm selection by clicking enter/double clicking with a mouse and measure the time I see the spectacle overlay/frozen x.org screencap+overlay.
3.Note: on CFQ I am able to interact with krunner while the overlay is still in place but I don't see the krunner, on BFQ the interaction with krunner will get registered after the overlay goes away.
4.Once the overlay goes away I stop the stopwatch on my phone and the measured time is the time I said it takes to take a screenshot(fullscreen screenshots aren't a problem), after the overlay goes away I can move around windows and interact with them.

The problems arouse once my lvm cache got filled, so I presumed the problem lays in hdd latency/my current drives setup.
dmsetup status:

data-cache_cdata: 0 249798656 linear 
data-cache_cmeta: 0 262144 linear 
data-root: 0 1942618112 cache 8 2940/32768 256 974950/975776 3913547 1531558 16415635 5956682 21964 21154 0 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 
data-root_corig: 0 1942618112 linear 

Last edited by kubast2 (2018-09-20 15:35:15)

brebs

Member

Registered: 2007-04-03

Posts: 3,742

Re: Balancing out latency and throughput with IO scheduler tunables/kernel

For qtbittorent hammering the drive, some suggestions:

1. Use e.g. noatime (or relatime) and commit=60, and nobarrier. The default commit of 5 seconds is paranoid, asumming your drives are reliable. Firefox runs sqlite, which fsyncs a lot, which kills performance. E.g. in /etc/fstab:

UUID=blah /  ext4  rw,relatime,nobarrier,commit=60	0 1

2. Run qbittorrent with nice and ionice, since it is a background task for you. E.g.:

nice ionice -c3 qbittorrent