Root process "/" using up all the CPU - fork bomb? [SOLVED]

bheadmaster · 2017-10-27 15:43:51

I have a weird problem that randomly happens every few weeks or so, after I wake up my laptop from hybernation - I get massive amounts of CPU load reported and my machine slows down to a crawl until it finally freezes completely.

Checking htop as root, moments before freezing, I managed to find the process that takes up all the CPU - and it's just called "/" - it doesn't react to any signals, I tried killing it with SIGKILL but nothing happened. I managed to take this screenshot before total freeze:

(I made it 'nice' just so I could buy myself some time - niceness was initially 0)

Here's the output from journalctl, after the moment I resumed from hybernation:

log.txt

Has anyone had similar experiences?

Last edited by bheadmaster (2017-12-20 17:51:20)

lo1 · 2017-10-27 16:46:18

Oct 27 17:07:22 archbox kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemctl:28664]

I'm seeing lots of this in your journaling. I may be clueless, but looks to me like some module isn't able to recover from hibernation properly. Try disabling the ones you added manually, if any, and retry. Otherwise, i really don't have a clue.

bheadmaster · 2017-10-27 17:38:09

Interesting observation.
Unfortunately, I don't remember if I've added any modules manually, it's been a long time since I installed my system. Is there any way to check it?

Also:

Oct 27 17:06:54 archbox kernel: note: systemctl[28664] exited with preempt_count 1

I see "systemctl" mentioned with the same PID as the process that caused freezing. I'm not quite good at reading systemd logs, but doesn't this mean the "systemctl" process is causing the freeze?

cfr · 2017-10-28 03:33:15

bheadmaster wrote:

Interesting observation.
Unfortunately, I don't remember if I've added any modules manually, it's been a long time since I installed my system. Is there any way to check it?

/etc/mkinitcpio.conf?

Also:
Oct 27 17:06:54 archbox kernel: note: systemctl[28664] exited with preempt_count 1
I see "systemctl" mentioned with the same PID as the process that caused freezing. I'm not quite good at reading systemd logs, but doesn't this mean the "systemctl" process is causing the freeze?

Probably not.

seth · 2017-10-28 12:05:15

Probably yes. It has quite some memory allocation issues - no idea why, but let's see whether we figure the cause. Output of "hostnamectl"
If the static hostname is n/a see https://wiki.archlinux.org/index.php/Ne … e_hostname

bheadmaster · 2017-10-28 18:02:56

cfr wrote:

/etc/mkinitcpio.conf?

Nothing meaningful there. I only remember I added some additional hooks:

HOOKS="base udev resume autodetect modconf block filesystems keyboard fsck"

but I doubt that they are in any way connected to the issue.

seth wrote:

Output of "hostnamectl"

Here it is:

Static hostname: archbox
Icon name: computer-laptop
Chassis: laptop
Machine ID: 1c261661d0104064a8f69b353c11f234
Boot ID: ddcb421065374f6d9bc6bdb45b07127c
Operating System: Arch Linux
Kernel: Linux 4.13.8-1-ARCH
Architecture: x86-64

seth · 2017-10-28 18:59:43

Nope - dead end (I speculated on issues with a transient only hostname)
Any idea what systemctl is doing there? (inspect /proc/$PID/cmdline when it happens next time)
Custom sleep/resume hooks maybe?

bheadmaster · 2017-10-28 19:46:45

I doubt it. I was very careful to follow wiki when I set up hibernation. Besides, it doesn't happen every time - it happens every once in a while, which makes it an even more mystery to me. I almost never reboot my machine except when I update the kernel and most of the time it wakes up from hibernation properly.

[lark@archbox ~]$ free -h
total used free shared buff/cache available
Mem: 3.8G 720M 118M 360M 3.0G 2.5G
Swap: 4.0G 0B 4.0G

I have enough swap for the whole image so I'm sure that's not an issue either.

seth wrote:

(inspect /proc/$PID/cmdline when it happens next time)

Will do. Hopefully it happens again sooner (rather than later) so I can bring you more information.

cfr · 2017-10-28 22:15:20

bheadmaster wrote:

I doubt it. I was very careful to follow wiki when I set up hibernation. Besides, it doesn't happen every time - it happens every once in a while, which makes it an even more mystery to me. I almost never reboot my machine except when I update the kernel and most of the time it wakes up from hibernation properly.

Ordering issue, maybe? systemd does things asynchronously by default ...

Have you compared a log from a resume without the issue to one with it?

Last edited by cfr (2017-10-28 22:15:57)

bheadmaster · 2017-10-29 07:46:43

cfr wrote:

Have you compared a log from a resume without the issue to one with it?

I don't even know where to begin when comparing logs since I'm not familiar at all with how systemd works.

But here is the journal log from this morning, in case someone else notices something meaningful:
proper_resume.txt

lo1 · 2017-10-30 22:34:52

bheadmaster wrote:

But here is the journal log from this morning, in case someone else notices something meaningful:
proper_resume.txt

Here's the same clueless guy than before, though now i'm sure that whatever watchdog or systemctl activity was causing the issue is gone in that proper_resume.
Next step, i would check if there's some differences between

systemctl list-units --state=active

both before and after hibernating, with and without experiencing the freeze. And DIY, i'm glad if i can help but you should do the reading part by yourself

bheadmaster · 2017-10-31 13:21:13

I see. You're right - it's my system and I should put more effort into analyzing logs.
That being said, I experienced something weird...

I had the same problem come up after waking up from hibernation, and I opened htop as root, but there wasn't any process taking up CPU - even though the CPU load was over 10 and system was unusably slow.

Still, I took a screenshot of the htop screen (like in the original post), dumped journalctl into a log file and started reading it, and found a line that said something about "kernel bug at address (null)". Since I already had the log dumped into a text file, and a screenshot of htop state, I called "systemctl poweroff" from the terminal. Then the screen went black but the computer was still on for a few minutes, and I had to turn it off physically.

Then I turned it on again - and the log file and screenshots were gone. I called journalctl again but there were only three lines left from the time I resumed it from hybernation:

Oct 31 13:54:05 archbox kernel: PM: done.
Oct 31 13:54:05 archbox kernel: Freezing user space processes ... (elapsed 0.001 seconds) done.
Oct 31 13:54:05 archbox kernel: OOM killer disabled.

And it doesn't make any sense. Maybe all the data got wiped when I forcibly turned off my machine?
Anyways, I clearly remember there was a kernel bug message in journalctl, and something about "usb disconnected or restarted". I hope someone makes sense out of all this, because I can't.

Maybe I fucked something up unknowingly. I'm tempted to do a system reinstall but it doesn't make sense if I don't figure out what's wrong...

seth · 2017-10-31 13:42:48

And it doesn't make any sense. Maybe all the data got wiped when I forcibly turned off my machine?

Quite likely hung on or before syncing the filesystems to disk (did you boot up with a fsck run?)

lo1 · 2017-10-31 13:56:29

Well, reinstalling is not productive but if you need your system for important stuff don't be too afraid to go that way (just my opinion).
Now, this is something that's really scratching my mind. Makes me think of a faulty swap partition/drive..or is it perhaps too little to store a full snap of your system? Let's see a

free -h

(while we wait for those systemctl active units to be compared..).
@seth is absolutely right, you're missing practically everything after the hibernation.

EDIT: swap size was already clarified, sorry for the redundancy.

Last edited by lo1 (2017-11-01 20:54:05)

bheadmaster · 2017-10-31 17:28:24

seth wrote:

did you boot up with a fsck run?

I found this in the journal, so I guess.

Oct 31 14:03:11 archbox systemd-fsck[303]: /dev/sda4: recovering journal
Oct 31 14:03:11 archbox systemd-fsck[303]: /dev/sda4: clean, 11653/27582464 files, 21757773/110299905 blocks
Oct 31 14:03:11 archbox systemd[1]: Started File System Check on /dev/disk/by-uuid/5d0f8a93-71a3-4ede-a9c7-cd739c9437b4.
Oct 31 14:03:11 archbox systemd[1]: Mounting /home...
Oct 31 14:03:12 archbox kernel: EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: data=ordered
Oct 31 14:03:12 archbox systemd[1]: Mounted /home.
Oct 31 14:03:12 archbox systemd-fsck[304]: fsck.fat 4.1 (2017-01-24)
Oct 31 14:03:12 archbox systemd-fsck[304]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Oct 31 14:03:12 archbox systemd-fsck[304]: Automatically removing dirty bit.
Oct 31 14:03:12 archbox systemd-fsck[304]: Performing changes.
Oct 31 14:03:12 archbox systemd-fsck[304]: /dev/sda1: 363 files, 13647/261628 clusters
Oct 31 14:03:12 archbox systemd[1]: Started File System Check on /dev/disk/by-uuid/677B-1DD0.
Oct 31 14:03:12 archbox systemd[1]: Mounting /boot...
Oct 31 14:03:12 archbox systemd[1]: Mounted /boot.
Oct 31 14:03:12 archbox systemd[1]: Reached target Local File Systems.
Oct 31 14:03:12 archbox systemd[1]: Starting Rebuild Dynamic Linker Cache...
Oct 31 14:03:12 archbox systemd[1]: Starting Create Volatile Files and Directories...
Oct 31 14:03:12 archbox systemd[1]: Starting Rebuild Journal Catalog...
Oct 31 14:03:12 archbox systemd[1]: Started Create Volatile Files and Directories.
Oct 31 14:03:12 archbox systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Oct 31 14:03:12 archbox systemd[1]: Started Rebuild Journal Catalog.
Oct 31 14:03:13 archbox systemd[1]: Started Update UTMP about System Boot/Shutdown.
Oct 31 14:03:13 archbox systemd[1]: Started Rebuild Dynamic Linker Cache.
Oct 31 14:03:13 archbox systemd[1]: Starting Update is Completed...
Oct 31 14:03:13 archbox systemd[1]: Started Update is Completed.
Oct 31 14:03:13 archbox systemd[1]: Reached target System Initialization.
Oct 31 14:03:13 archbox systemd[1]: Started Daily man-db cache update.
Oct 31 14:03:13 archbox systemd[1]: Started Daily Cleanup of Temporary Directories.
Oct 31 14:03:13 archbox systemd[1]: Started Daily rotation of log files.
Oct 31 14:03:13 archbox systemd[1]: Listening on D-Bus System Message Bus Socket.
Oct 31 14:03:13 archbox systemd[1]: Reached target Sockets.
Oct 31 14:03:13 archbox systemd[1]: Reached target Basic System.
Oct 31 14:03:13 archbox systemd[1]: Started D-Bus System Message Bus.
Oct 31 14:03:14 archbox systemd[1]: Starting Login Service...
Oct 31 14:03:14 archbox systemd[1]: Started Daily verification of password and group files.
Oct 31 14:03:14 archbox systemd[1]: Reached target Timers.
Oct 31 14:03:14 archbox systemd[1]: Starting Network Manager...
Oct 31 14:03:14 archbox systemd[1]: Started Login Service.

lo1 wrote:

Let's see a
free -h
(while we wait for those systemctl active units to be compared..).

I've posted it in #8, but here:

[lark@archbox ~]$ free -h
total used free shared buff/cache available
Mem: 3.8G 670M 120M 218M 3.0G 2.7G
Swap: 4.0G 0B 4.0G

Sorry about the lack systemctl active units comparison. I got a little distracted by all the weird stuff happening. I'll try to do it tomorrow, hopefully manage to get the freeze in the process.

lo1 · 2017-10-31 18:04:54

Sorry, i completely missed that. Don't mind for the delay, we'll see it tomorrow (and hopefully i'll come up with more useful hints)

bheadmaster · 2017-11-01 17:23:50

Didn't experience any freezing tonight. Here are the "systemctl --state=active" logs:
before_hibernation
after_hibernation
However, just to save you from searching for differences:

diff before_hibernation_-_no_freeze.txt after_hibernation_-_no_freeze.txt

gives me nothing, so they are identical.

I'll get the other logs as soon as I get another freeze.

lo1 · 2017-11-01 20:51:29

That is a good sign i guess. While we wait, have a look here, which deals mostly with problem with encryption + hibernation. Not related, but you'll get some ideas for sure. Also, try to remember the most recent changes which could have fucked up the hibernation process, i've got a sense this is simpler than it seems but we're not isolating the issue

bheadmaster · 2017-12-20 17:50:59

Here's some update, just for the final info.

Ever since I stopped using i3lock-blur from the AUR and installed vanilla i3lock from the repo, these weird freezes have stopped happening.
This was in my i3 config:

set $Locker i3lock --fuzzy --no-unlock-indicator && sleep 2
[...]
bindsym h exec --no-startup-id $Locker && systemctl hibernate, mode "default"

I guess it somehow fscked with my hybernation.
The i3lock-blur seems abandoned anyway, so I guess this is it.

Arch Linux

#1 2017-10-27 15:43:51

Root process "/" using up all the CPU - fork bomb? [SOLVED]

#2 2017-10-27 16:46:18

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#3 2017-10-27 17:38:09

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#4 2017-10-28 03:33:15

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#5 2017-10-28 12:05:15

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#6 2017-10-28 18:02:56

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#7 2017-10-28 18:59:43

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#8 2017-10-28 19:46:45

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#9 2017-10-28 22:15:20

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#10 2017-10-29 07:46:43

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#11 2017-10-30 22:34:52

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#12 2017-10-31 13:21:13

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#13 2017-10-31 13:42:48

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#14 2017-10-31 13:56:29

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#15 2017-10-31 17:28:24

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#16 2017-10-31 18:04:54

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#17 2017-11-01 17:23:50

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#18 2017-11-01 20:51:29

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

#19 2017-12-20 17:50:59

Re: Root process "/" using up all the CPU - fork bomb? [SOLVED]

Board footer