You are not logged in.
- Topics: Active | Unanswered
#1 2022-06-28 03:04:37
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
[SOLVED] Periodic CPU spikes by high-priority short-living process
Hi.
It's been about 18 months on Linux / ArchLinux.
The start
After a successful [pacman -Syu] 3-4 weeks ago, I noticed frequent glitches in VLC's video that took 5-10 seconds. On gnome-system-monitor, the 10 minute CPU chart showed periodic full spikes of all 4 logical cores with intervals of about 57 seconds.
In the Processes tab, it "seemed" that 4 systemd processes have full CPU usage in the same moment, but no process had an accumulated CPU time that could stand for that much of processing. So it seemed like a recurring systemd job whose process starts and ends. Nothing was logged to journalctl before/after every spike.
Exponential growth
It seemed like a techincal problem, but after a couple of days, the interval halved (didn't measure exactly), and in a few days, it halved again.
Other possible symptoms
Another issue at the time - that I didn't (and still don't) know if was related - was something like this post except that the process in the final parentheses was "winedevice.exe".
Did a research on "winedevice.exe" and 2 sites were calling it a miner. Panicked and rebooted. Booted from ISO and removed wine in chroot - for now.
Trying to use clamav
The boot ISO's temporary root FS was too small to install clamav (Why not to use more RAM when it's there?). Then I tried to do that in single-user mode, but clamav's definition server denied me from downloading until the next day!! (they used some word for it) and I couldn't wait.
Re-entering the desktop
The spikes did not happen until at least many hours, and then they took a second or less and didn't make a full spike, though still causing glitches in video.
Trying to find the process
I couldn't see it in top or htop, with the default delay or a delay of 0.2 seconds.
Ran bash as root, gave it a "Very high priority",and ran this:
while true; do
date +%H:%M:%S.%N >> p
ps -aux --sort=-%cpu|head -n6 >> p
doneand I could see that 'ps' is run about every 0.04 seconds, but when the spike comes, there is a 3-4 seconds stop (the spikes had become longer again). That thing has a crazy priority.
Trying to use Kaspersky Rescue Disk
It's cryptsetup said it can't open LUKS2.
Re-entering the desktop again
I have not noticed the spikes in the CPU chart, nor the glitches in the video - yet.
Thanks in advance.
Last edited by truth_believer (2022-07-05 16:31:50)
Offline
#2 2022-06-28 06:59:59
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Some kernel thread?
Have you looked into dmesg or the system journal?
the process in the final parentheses was "winedevice.exe"
On your system (cause it's not in the other thread)?
% pacman -F winedevice.exe
multilib/wine 7.11-1
usr/lib/wine/x86_64-windows/winedevice.exe
usr/lib32/wine/i386-windows/winedevice.exe
multilib/wine-staging 7.11-1
usr/lib/wine/x86_64-windows/winedevice.exe
usr/lib32/wine/i386-windows/winedevice.exeBTC mining is past anyway - and it would occupy a lot of CPU all the time on low priority to not be notable.
For symptom clarification: did you update, *not* reboot, then had that frequent problem and then rebooted (being afraid that somebody abuses your system to produce a sham "currency") and then the original symptoms went away (isolated cpu spikes could have all sorts of reasons)
Offline
#3 2022-06-28 09:00:56
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Some kernel thread?
I'm not sure I get it. I guess you're suggesting that I should post a kernel thread, right?
Have you looked into dmesg
Don't know how.
or the system journal?
You mean checking the whole journalctl apart from that I said "Nothing was logged to journalctl before/after every spike."?
I've searched journalctl for the expression "error" but more than that, I don't know how. 
the process in the final parentheses was "winedevice.exe"
On your system (cause it's not in the other thread)?
Yes.
Could find it from firefox history; this was my repeating journalctl line:
dbus-daemon[794]: [system] Rejected: destination has a full message queue, 0 matched rules; type="signal", sender=":1.3" (uid=0 pid=1 comm="/sbin/init") interface="org.freedesktop.DBus.Properties" member="PropertiesChanged" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (uid=60159 pid=47419 comm="C:\windows\system32\winedevice.exe ")% pacman -F winedevice.exe multilib/wine 7.11-1 usr/lib/wine/x86_64-windows/winedevice.exe usr/lib32/wine/i386-windows/winedevice.exe multilib/wine-staging 7.11-1 usr/lib/wine/x86_64-windows/winedevice.exe usr/lib32/wine/i386-windows/winedevice.exe
I'm not sure I get it.
I guess you mean "winedevice.exe" is a legitimate wine program and I shouldn't worry about that, right?
BTC mining is past anyway - and it would occupy a lot of CPU all the time on low priority to not be notable.
That's reasonable.
I could find again the site that said it's a miner here https://howtofix.guide/winedevice-exe-file/ and I think it is shady.
Sorry.
For symptom clarification: did you update, *not* reboot, then had that frequent problem and then rebooted (being afraid that somebody abuses your system to produce a sham "currency") and then the original symptoms went away (isolated cpu spikes could have all sorts of reasons)
I do prefer to reboot after update, and I sort of remember having rebooted in that particular time too, though it is possible that I have forgotten it.
But I'm sure the first time that I encountered the issue, the second thing I did was to reboot, after having tried to identify the cause first. And I'm quite sure that the spikes didn't go away.
Last edited by truth_believer (2022-06-28 09:02:59)
Offline
#4 2022-06-28 10:44:21
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
The spikes are back, and smaller than 20%, yet I noticed them from glitches in the video!
I knew I forgot something. That was, to particularly emphasize on this:
didn't make a full spike, though still causing glitches in video.
This was the strangest part for me, that the CPU was not fully used, yet VLC or ps would have problem utilizing the rest of the CPU time when the spike was happening!
And, is this PID of '?' normal?
NetHogs version 0.8.7
PID USER PROGRAM DEV SENT RECEIVED
1191 mohamm.. /usr/lib/firefox/firefox enp9s0 0.011 0.012 KB/sec
? root unknown TCP 0.000 0.000 KB/sec
TOTAL 0.011 0.012KB/secIt communicates something once in a while.
Last edited by truth_believer (2022-06-28 10:48:57)
Offline
#5 2022-06-28 14:11:34
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
I guess you're suggesting that I should post a kernel thread, right?
No, a kernel thread like an anonymous process run by the kernel, try "top -H"
Don't know how.
https://man.archlinux.org/man/core/util … dmesg.1.en
the whole journalctl apart from that I said "Nothing was logged to journalctl before/after every spike."
Unless your UID is in an admin group, "journalctl" will only post the session journal. In this case "sudo journalctl -b" would show you the entire system journal of the boot.
I've searched journalctl for the expression "error"
It's unlikely that this will show up as "error", you're looking for anything suspicious that time-aligns w/ the perceived CPU spikes.
dbus-daemon[794]: [system] Rejected: destination has a full message queue
means that (for winedevice "likely") the session bus is jammed.
If that's a frequent issue, "dbus-monitor" might hint the offender.
I guess you mean "winedevice.exe" is a legitimate wine program and I shouldn't worry about that, right?
I'm not saying that it's not malicious for sure, but chances for that are quite low. And showing up in the dbus jam, it's an innocent messenger.
And, is this PID of '?' normal?
Either you've insufficient rights to inspect the PID attached to the socket, or this socket belongs to the kernel (some NFS/CIFS mount etc)
Offline
#6 2022-07-01 16:06:40
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Thanks again.
No, a kernel thread like an anonymous process run by the kernel, try "top -H"
Here are the top 8 threads of 4 subsequent spikes:
────────────────────────────────────────────────────────────────────────────────────────────────────
Spike 1
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:47.912
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
861 mohammad 20 0 21360 14128 8936 R 23.5 0.2 2:19.47 systemd
24661 root 20 0 19492 12224 8932 R 23.5 0.2 0:22.05 systemd
1 root 20 0 170832 16524 9276 R 17.6 0.2 5:57.17 systemd
174 root 20 0 0 0 0 S 5.9 0.0 0:06.09 usb-storage
39712 mohammad 20 0 658980 146876 98800 S 5.9 1.9 6:55.11 xfce4-terminal
48431 mohammad 20 0 63564 18132 13368 S 5.9 0.2 6:12.75 aria2c
80418 root 20 0 12312 4932 4144 R 5.9 0.1 0:00.02 top
80421 root 20 0 32572 6424 3520 S 5.9 0.1 0:00.01 systemd-udevd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:48.122
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:57.39 systemd
861 mohammad 20 0 21360 14128 8936 R 99.9 0.2 2:19.68 systemd
668 dbus 20 0 9876 5136 3680 R 70.6 0.1 11:05.07 dbus-daemon
674 root 20 0 48568 7948 7000 R 23.5 0.1 3:23.39 systemd-logind
24661 root 20 0 19492 12224 8932 S 11.8 0.2 0:22.12 systemd
80425 root 20 0 12312 4776 3992 R 11.8 0.1 0:00.04 top
681 root 20 0 15936 9360 8248 S 5.9 0.1 0:06.30 systemd-homed
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:48.350
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:57.64 systemd
668 dbus 20 0 9876 5136 3680 R 90.0 0.1 11:05.32 dbus-daemon
674 root 20 0 48568 7948 7000 S 25.0 0.1 3:23.46 systemd-logind
861 mohammad 20 0 21360 14128 8936 S 25.0 0.2 2:19.79 systemd
39712 mohammad 20 0 658980 146876 98800 S 5.0 1.9 6:55.12 xfce4-terminal
80429 root 20 0 12312 4872 4084 R 5.0 0.1 0:00.04 top
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:48.588
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:57.85 systemd
668 dbus 20 0 9876 5136 3680 R 94.1 0.1 11:05.51 dbus-daemon
674 root 20 0 48568 7948 7000 S 23.5 0.1 3:23.51 systemd-logind
30797 mohammad 20 0 158792 6168 5488 S 5.9 0.1 0:03.54 gdbus
39473 mohammad 20 0 617260 56500 42120 S 5.9 0.7 0:44.20 Thunar
80433 root 20 0 12312 4772 3988 R 5.9 0.1 0:00.02 top
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:48.787
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:58.04 systemd
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:05.71 dbus-daemon
674 root 20 0 48568 7948 7000 S 23.5 0.1 3:23.55 systemd-logind
48431 mohammad 20 0 64108 18392 13368 S 5.9 0.2 6:12.77 aria2c
80437 root 20 0 12312 4844 4060 R 5.9 0.1 0:00.02 top
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:48.985
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:05.91 dbus-daemon
674 root 20 0 48568 7948 7000 S 35.3 0.1 3:23.62 systemd-logind
39454 mohammad 20 0 686668 92888 73804 S 5.9 1.2 2:21.32 xfwm4
48431 mohammad 20 0 64244 18656 13368 S 5.9 0.2 6:12.78 aria2c
80441 root 20 0 12312 4788 4004 R 5.9 0.1 0:00.02 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.06 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:49.186
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:06.12 dbus-daemon
674 root 20 0 48568 7948 7000 R 27.8 0.1 3:23.68 systemd-logind
80445 root 20 0 12312 4932 4144 R 11.1 0.1 0:00.04 top
39712 mohammad 20 0 658980 146876 98800 S 5.6 1.9 6:55.14 xfce4-terminal
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.06 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:49.401
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:06.32 dbus-daemon
674 root 20 0 48568 7948 7000 S 35.3 0.1 3:23.75 systemd-logind
80449 root 20 0 12312 4928 4144 R 11.8 0.1 0:00.03 top
39345 root 20 0 522060 195812 170940 S 5.9 2.5 6:48.09 Xorg
39712 mohammad 20 0 658980 146876 98800 S 5.9 1.9 6:55.15 xfce4-terminal
48431 mohammad 20 0 63564 18132 13368 S 5.9 0.2 6:12.79 aria2c
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.06 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:10:49.601
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 S 16.7 0.1 11:06.38 dbus-daemon
39345 root 20 0 522060 195812 170940 S 5.6 2.5 6:48.10 Xorg
39712 mohammad 20 0 658980 146876 98800 S 5.6 1.9 6:55.16 xfce4-terminal
48431 mohammad 20 0 63700 18132 13368 S 5.6 0.2 6:12.80 aria2c
80453 root 20 0 12312 4776 3988 R 5.6 0.1 0:00.04 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.06 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
Spike 2
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:44.401
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
861 mohammad 20 0 21360 14128 8936 R 99.9 0.2 2:19.99 systemd
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:58.33 systemd
668 dbus 20 0 9876 5136 3680 R 35.3 0.1 11:06.45 dbus-daemon
24661 root 20 0 19492 12224 8932 S 29.4 0.2 0:22.21 systemd
39712 mohammad 20 0 658980 146876 98800 S 11.8 1.9 6:57.28 xfce4-terminal
674 root 20 0 48568 7948 7000 R 5.9 0.1 3:23.77 systemd-logind
39345 root 20 0 533964 205448 180568 S 5.9 2.6 6:49.69 Xorg
39473 mohammad 20 0 617260 56500 42120 S 5.9 0.7 0:44.40 Thunar
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:44.651
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 90.9 0.2 5:58.60 systemd
668 dbus 20 0 9876 5136 3680 R 90.9 0.1 11:06.73 dbus-daemon
861 mohammad 20 0 21360 14128 8936 S 68.2 0.2 2:20.20 systemd
674 root 20 0 48568 7948 7000 R 31.8 0.1 3:23.87 systemd-logind
81590 root 20 0 12312 4964 4176 R 9.1 0.1 0:00.04 top
48431 mohammad 20 0 64116 18392 13368 S 4.5 0.2 6:16.05 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:44.916
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:06.95 dbus-daemon
1 root 20 0 170832 16524 9276 R 94.4 0.2 5:58.81 systemd
674 root 20 0 48568 7948 7000 S 33.3 0.1 3:23.94 systemd-logind
81594 root 20 0 12312 4936 4152 R 11.1 0.1 0:00.04 top
29 root 20 0 0 0 0 S 5.6 0.0 0:02.95 ksoftirqd/2
48431 mohammad 20 0 64400 18656 13368 S 5.6 0.2 6:16.06 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:45.136
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 94.4 0.1 11:07.17 dbus-daemon
1 root 20 0 170832 16524 9276 S 50.0 0.2 5:58.95 systemd
674 root 20 0 48568 7948 7000 S 38.9 0.1 3:24.02 systemd-logind
39712 mohammad 20 0 658980 146876 98800 S 11.1 1.9 6:57.32 xfce4-terminal
81598 root 20 0 12312 4932 4144 R 11.1 0.1 0:00.03 top
48431 mohammad 20 0 63564 18132 13368 S 5.6 0.2 6:16.07 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:45.348
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:07.38 dbus-daemon
674 root 20 0 48568 7948 7000 S 33.3 0.1 3:24.09 systemd-logind
81602 root 20 0 12312 4776 3992 R 22.2 0.1 0:00.05 top
39712 mohammad 20 0 658980 146876 98800 S 11.1 1.9 6:57.34 xfce4-terminal
48431 mohammad 20 0 63844 18132 13368 S 5.6 0.2 6:16.08 aria2c
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.95 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:45.562
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:07.58 dbus-daemon
674 root 20 0 48568 7948 7000 S 35.3 0.1 3:24.16 systemd-logind
48431 mohammad 20 0 64116 18392 13368 S 11.8 0.2 6:16.10 aria2c
81606 root 20 0 12312 4780 3992 R 5.9 0.1 0:00.02 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.95 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:11:45.759
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 S 50.0 0.1 11:07.70 dbus-daemon
674 root 20 0 48568 7948 7000 S 16.7 0.1 3:24.20 systemd-logind
39712 mohammad 20 0 658980 146876 98800 S 11.1 1.9 6:57.36 xfce4-terminal
81610 root 20 0 12312 4868 4084 R 11.1 0.1 0:00.03 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:58.95 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
Spike 3
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:40.585
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
861 mohammad 20 0 21360 14128 8936 R 47.1 0.2 2:20.28 systemd
24661 root 20 0 19492 12224 8932 R 47.1 0.2 0:22.29 systemd
1 root 20 0 170832 16524 9276 R 41.2 0.2 5:59.11 systemd
82726 root 20 0 12312 4952 4168 R 17.6 0.1 0:00.04 top
1039 root 20 0 393372 13280 9836 S 5.9 0.2 0:06.01 udisksd
48431 mohammad 20 0 63564 18132 13368 S 5.9 0.2 6:19.22 aria2c
82724 root 20 0 32572 6424 3520 S 5.9 0.1 0:00.01 systemd-udevd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:40.798
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:59.33 systemd
861 mohammad 20 0 21360 14128 8936 R 94.4 0.2 2:20.49 systemd
668 dbus 20 0 9876 5136 3680 R 83.3 0.1 11:07.86 dbus-daemon
674 root 20 0 48568 7948 7000 R 22.2 0.1 3:24.24 systemd-logind
39712 mohammad 20 0 658980 146876 98800 S 22.2 1.9 6:59.88 xfce4-terminal
82732 root 20 0 12312 4940 4156 R 16.7 0.1 0:00.05 top
39345 root 20 0 518256 188256 163376 S 11.1 2.4 6:50.23 Xorg
48431 mohammad 20 0 63972 18392 13368 S 11.1 0.2 6:19.24 aria2c
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:41.020
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 R 99.9 0.2 5:59.56 systemd
668 dbus 20 0 9876 5136 3680 R 94.4 0.1 11:08.07 dbus-daemon
861 mohammad 20 0 21360 14128 8936 S 38.9 0.2 2:20.60 systemd
674 root 20 0 48568 7948 7000 R 33.3 0.1 3:24.31 systemd-logind
48431 mohammad 20 0 64244 18656 13368 S 11.1 0.2 6:19.26 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:41.242
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 170832 16524 9276 S 94.4 0.2 5:59.77 systemd
668 dbus 20 0 9876 5136 3680 R 94.4 0.1 11:08.28 dbus-daemon
674 root 20 0 48568 7948 7000 R 33.3 0.1 3:24.38 systemd-logind
82740 root 20 0 12312 4776 3992 R 11.1 0.1 0:00.04 top
48431 mohammad 20 0 63564 18132 13368 S 5.6 0.2 6:19.27 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:41.455
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:08.49 dbus-daemon
674 root 20 0 48568 7948 7000 S 31.2 0.1 3:24.43 systemd-logind
82744 root 20 0 12312 4952 4168 R 12.5 0.1 0:00.03 top
23 root 20 0 0 0 0 S 6.2 0.0 0:03.29 ksoftirqd/1
39712 mohammad 20 0 658980 146876 98800 S 6.2 1.9 6:59.89 xfce4-terminal
48431 mohammad 20 0 63828 18132 13368 S 6.2 0.2 6:19.28 aria2c
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:59.77 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:41.656
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 94.1 0.1 11:08.68 dbus-daemon
674 root 20 0 48568 7948 7000 S 35.3 0.1 3:24.51 systemd-logind
39473 mohammad 20 0 617260 56500 42120 S 5.9 0.7 0:44.61 Thunar
39712 mohammad 20 0 658980 146876 98800 S 5.9 1.9 6:59.90 xfce4-terminal
48431 mohammad 20 0 64096 18392 13368 S 5.9 0.2 6:19.29 aria2c
82748 root 20 0 12312 4940 4152 R 5.9 0.1 0:00.02 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:59.77 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:41.859
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:08.89 dbus-daemon
674 root 20 0 48568 7948 7000 S 27.8 0.1 3:24.57 systemd-logind
82754 root 20 0 12312 4776 3992 R 11.1 0.1 0:00.03 top
48431 mohammad 20 0 64368 18656 13368 S 5.6 0.2 6:19.30 aria2c
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:59.77 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:12:42.067
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 S 52.9 0.1 11:09.01 dbus-daemon
674 root 20 0 48568 7948 7000 S 11.8 0.1 3:24.60 systemd-logind
48431 mohammad 20 0 63700 18132 13368 S 11.8 0.2 6:19.32 aria2c
39712 mohammad 20 0 658980 146876 98800 S 5.9 1.9 6:59.93 xfce4-terminal
82758 root 20 0 12312 4792 4004 R 5.9 0.1 0:00.02 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 5:59.77 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
Spike 4
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:36.980
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
861 mohammad 20 0 21360 14128 8936 R 82.4 0.2 2:20.74 systemd
1 root 20 0 170832 16524 9276 R 76.5 0.2 5:59.99 systemd
24661 root 20 0 19492 12224 8932 S 47.1 0.2 0:22.40 systemd
83868 root 20 0 12312 4772 3988 R 11.8 0.1 0:00.02 top
681 root 20 0 15936 9360 8248 S 5.9 0.1 0:06.83 systemd-homed
1036 mohammad 20 0 356116 12252 10176 S 5.9 0.2 0:04.19 gvfs-udisks2-vo
1048 root 20 0 393372 13280 9836 S 5.9 0.2 0:00.13 probing-thread
48431 mohammad 20 0 63564 18132 13368 R 5.9 0.2 6:24.18 aria2c
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:37.182
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:09.22 dbus-daemon
861 mohammad 20 0 21360 14128 8936 R 99.9 0.2 2:20.96 systemd
1 root 20 0 170832 16524 9276 R 94.4 0.2 6:00.19 systemd
674 root 20 0 48568 7948 7000 R 33.3 0.1 3:24.67 systemd-logind
83874 root 20 0 12312 4932 4144 R 16.7 0.1 0:00.05 top
48431 mohammad 20 0 63988 18392 13368 S 5.6 0.2 6:24.20 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:37.422
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:09.45 dbus-daemon
1 root 20 0 170832 16524 9276 R 94.4 0.2 6:00.42 systemd
674 root 20 0 48568 7948 7000 S 33.3 0.1 3:24.76 systemd-logind
83878 root 20 0 12312 4776 3988 R 16.7 0.1 0:00.04 top
39712 mohammad 20 0 658980 146876 98800 S 11.1 1.9 7:01.92 xfce4-terminal
48431 mohammad 20 0 64404 18656 13368 R 11.1 0.2 6:24.22 aria2c
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:37.647
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:09.67 dbus-daemon
1 root 20 0 170832 16524 9276 R 94.4 0.2 6:00.63 systemd
674 root 20 0 48568 7948 7000 R 33.3 0.1 3:24.83 systemd-logind
23 root 20 0 0 0 0 S 5.6 0.0 0:03.31 ksoftirqd/1
30795 mohammad 20 0 158792 6168 5488 S 5.6 0.1 0:05.14 gvfsd-metadata
39473 mohammad 20 0 617260 56500 42120 S 5.6 0.7 0:44.82 Thunar
83882 root 20 0 12312 4860 4076 R 5.6 0.1 0:00.03 top
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:37.863
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 94.4 0.1 11:09.88 dbus-daemon
674 root 20 0 48568 7948 7000 S 38.9 0.1 3:24.90 systemd-logind
681 root 20 0 15936 9360 8248 S 5.6 0.1 0:06.84 systemd-homed
48431 mohammad 20 0 63968 18392 13368 S 5.6 0.2 6:24.25 aria2c
83886 root 20 0 12312 4860 4076 R 5.6 0.1 0:00.03 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 6:00.66 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:38.069
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 R 99.9 0.1 11:10.09 dbus-daemon
674 root 20 0 48568 7948 7000 S 33.3 0.1 3:24.96 systemd-logind
83890 root 20 0 12312 4936 4152 R 11.1 0.1 0:00.03 top
39712 mohammad 20 0 658980 146876 98800 S 5.6 1.9 7:01.93 xfce4-terminal
48431 mohammad 20 0 64244 18656 13368 S 5.6 0.2 6:24.26 aria2c
1 root 20 0 170832 16524 9276 S 0.0 0.2 6:00.66 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
────────────────────────────────────────────────────────────────────────────────────────────────────
19:13:38.280
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 dbus 20 0 9876 5136 3680 S 88.2 0.1 11:10.27 dbus-daemon
674 root 20 0 48568 7948 7000 S 23.5 0.1 3:25.01 systemd-logind
48431 mohammad 20 0 63564 18132 13368 S 5.9 0.2 6:24.27 aria2c
83894 root 20 0 12312 4920 4132 R 5.9 0.1 0:00.02 top
1 root 20 0 170832 16524 9276 S 0.0 0.2 6:00.66 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
────────────────────────────────────────────────────────────────────────────────────────────────────Using 'dmeg -T' I can see there is no messages for so many minutes while the spikes have been happening every minute.
Should I be able to interpret some of the earlier messages?
dbus-daemon[794]: [system] Rejected: destination has a full message queue
means that (for winedevice "likely") the session bus is jammed.
If that's a frequent issue, "dbus-monitor" might hint the offender.
I've uninstalled wine for now to rule it out, and the "Rejected" messages have gone as well.
Will look into it later.
Either you've insufficient rights to inspect the PID attached to the socket, or this socket belongs to the kernel (some NFS/CIFS mount etc)
I've used root. Can that be insufficient?
I have no network mount; only an SSH server which didn't run at the moment, and the systemd ntp client.
Now there was this one:
PID USER PROGRAM DEV SENT RECEIVED
? root 192.168.1.2:50280-142.250.185.228:443 0.000 0.000 KB/sec
? root unknown TCP 0.000 0.000 KB/secThe IP is located at 8 different places, all from Google ISP.
But what does root have, that does a communication with this?
Last edited by truth_believer (2022-07-01 16:35:17)
Offline
#7 2022-07-01 16:31:07
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
There's quite a pattern in your top's involving systemd, logind and dbus-daemon which fits the "dbus-daemon[794]: [system] Rejected: destination has a full message queue"
Please post your entire system journal,
sudo journalctl -b | curl -F 'file=@-' 0x0.stOffline
#8 2022-07-01 16:38:02
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
There's quite a pattern in your top's involving systemd, logind and dbus-daemon which fits the "dbus-daemon[794]: [system] Rejected: destination has a full message queue"
Exactly.
Please post your entire system journal,
sudo journalctl -b | curl -F 'file=@-' 0x0.st
Here: http://0x0.st/oQi9.txt
Last edited by truth_believer (2022-07-01 16:38:26)
Offline
#9 2022-07-01 16:51:48
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
My money is on systemd-homed - do you use that <uglyswearing> intentionally?
Offline
#10 2022-07-01 17:25:49
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
My money is on systemd-homed - do you use that <uglyswearing> intentionally?
As I remember, after installing Arch I went for xfce and knew lightdm from there, and it couldn't log in, giving some pam-related Freedesktop.org errors that the net said had to do with systemd-homed.
It has caused me headaches multiple times over simple obvious things like login from console. I always wondered how everybody is tolerating it. So they don't?
Offline
#11 2022-07-01 17:35:49
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
systemd-homed will yell some pam error at you, which you can silence by unlisting the pam module
They're however just spam and will not cause login issues.
If you don't really intend to use whatever the systemd-homed feature is that actually works in a reasonable way: disable it and see what impact that has on the CPU spikes.
Offline
#12 2022-07-04 09:05:42
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Thanks.
Finally I could get some time.
Disabling systemd-homed didn't change the spikes.
Disabling systemd-udevd did. (of course along with disabling the 2 sockets who trigger it.)
Saw that coming before a spike, though not before every one. Probably not in the top 8 threads?
I had only 1 udev rule, which I removed for now.
Even before removing the rule, starting systemd-udevd again didn't make the spikes appear.
---
I am totally interested in disabling systemd-homed but couldn't find anything in the wiki. About unlisting the pam module as well.
Would you please point me?
---
Is it advisable to completely abandon systemd, given it is the only init system officially supported by arch?
Last edited by truth_believer (2022-07-04 09:59:15)
Offline
#13 2022-07-04 13:01:58
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
/etc/pam.d/system-auth - #comment the lines that have pam_systemd_home.so in it
And then just disable the service again.
I had only 1 udev rule, which I removed for now.
Can you post it?
Offline
#14 2022-07-04 17:53:23
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Can you post it?
/etc/udev/rules.d/50-usb_power_save.rules
--------------------------------
# blacklist for usb autosuspend
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="258a", ATTR{idProduct}=="0001", ATTR{power/autosuspend}="-1"Last edited by truth_believer (2022-07-04 17:54:06)
Offline
#15 2022-07-04 20:01:13
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
/etc/pam.d/system-auth - #comment the lines that have pam_systemd_home.so in it
And then just disable the service again.
Did it.
First forgot to create my user again by adduser, so encountered wrong password.
Then didn't think about the home dir's ownership, so was jumping out after login.
Thank you for telling me this. I loved to have a traditional user and it's benefits.
Offline
#16 2022-07-04 20:14:34
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Regarding this:
Is it advisable to completely abandon systemd, given it is the only init system officially supported by arch?
I got a tooth-breaking answer from https://archlinux.org/news/end-of-initscripts-support/.
Offline
#17 2022-07-04 20:36:43
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Your udev rule is unsupicious enough 
Have the spikes meanwhile manifested again?
Offline
#18 2022-07-05 04:11:14
- justinnhli
- Member
- Registered: 2009-04-05
- Posts: 10
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
I am facing something similar. Feel free to tell me to start a new thread, but I've narrowed my problem to dbus getting spammed by signals. I confirmed it by running
dbus-monitor --system --profile, which gave output like: https://pastebin.com/t3WkTG4Y. I'm still not sure what's causing it though.
Offline
#19 2022-07-05 06:31:23
- seth
- Member
- From: Don't DM me only for attention
- Registered: 2012-09-03
- Posts: 74,314
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Do you get this w/o a desktop session (a console login *only*)?
Offline
#20 2022-07-05 09:14:10
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Thanks.
Your udev rule is unsupicious enough
Have the spikes meanwhile manifested again?
In the same interval of 57.x seconds, I see small spikes of 5 to 10% from all cores.
i.e. the same pattern.
So yes.
They were sometimes very small before, too.
Maybe the rules shipped with a package cause it.
Used Debug output and restarted systemd-udevd.
Last edited by truth_believer (2022-07-05 09:33:48)
Offline
#21 2022-07-05 09:45:45
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Used Debug output and restarted systemd-udevd.
Got it: http://0x0.st/oQ2T.txt
Happens every 56 or 57 seconds.
Each one starts with:
sdc: Device is queued (SEQNUM=...
Validate module indexNot to be confused with the other that comes in the middle of each one:
sdc: Device is queued (SEQNUM=...
sdc: SEQNUM=???? blocked---
And I have no sdc.
And when I plug a USB flash drive, it is assigned sdd.
The SD Card Reader I guess.
---
Which of the rules to raid?
Last edited by truth_believer (2022-07-05 10:12:15)
Offline
#22 2022-07-05 10:09:31
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Did
echo 1 > /sys/block/sdx/device/deletefor sdc, and systemd-udevd became silent.
Will try to make it permanent.
Thank you very much seth for your help. Will bother you on other topics.
Last edited by truth_believer (2022-07-05 10:16:44)
Offline
#23 2022-07-05 12:37:02
- truth_believer
- Member
- Registered: 2022-06-15
- Posts: 17
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Made permanent.
To possibly help others:
$ realpath /sys/block/sdx/device
gives a persistent device path. Assuming it as ????????, we make a rule at
/etc/udev/rules.d/50-disable-this-thing.rules
containing
- -
ENV{DEVPATH}=="????????", RUN+="/bin/sh -c 'echo 1 > ????????/delete'"
- -
then
# udevadm control --reload
Offline
#24 2022-07-05 14:49:57
- ewaller
- Administrator
- From: Pasadena, CA
- Registered: 2009-07-13
- Posts: 20,612
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
Nice. Be sure to edit your first post and change the title be prepending "[SOLVED]" to the thread title. You may need to truncate the title to get that to fit. But this thread is worth tagging.
Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way
Offline
#25 2022-07-05 18:09:55
- justinnhli
- Member
- Registered: 2009-04-05
- Posts: 10
Re: [SOLVED] Periodic CPU spikes by high-priority short-living process
This solved my problem too, thanks!
In case it's useful to anyone, my permanent fix wasn't exactly the same as truth_believer's. Specifically, the DEVPATH and the block path different. As with truth_believer, I started by finding the persistent device path:
$ realpath /sys/block/sdb/device
/sys/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host6/target6:0:0/6:0:0:0But I then had to find the udev DEVPATH with
$ udevadm info --query=all --name=/dev/sdb | grep DEVPATH
E: DEVPATH=/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host6/target6:0:0/6:0:0:0/block/sdbMy final udev rule is:
$ cat /etc/udev/rules.d/50-disable-dbus-spam.rules
ENV{DEVPATH}=="/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host6/target6:0:0/6:0:0:0/block/sdb", RUN+="/bin/sh -c 'echo 1 > /sys/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host6/target6:0:0/6:0:0:0/delete'"Before I can finally reload udev:
# udevadm control --reloadLast edited by justinnhli (2022-07-05 18:10:47)
Offline