Arch Linux

vic_acid

Member

Registered: 2016-12-05

Posts: 13

Periodic high CPU bursts for /sbin/init and systemd --user...

I'm running into an odd issue here that I hope someone has seen before or can help me diagnose:

I'm running Arch on a Zotac z-box miniPC as a home gateway/router/firewall. Celeron N3150 CPU (Braswell era, IIRC). 8GB memory, 2GB zswap partitions, 2x gigabit LAN ports (ext and int facing).

About every 49-50 seconds, I see `/sbin/init`, `systemd --user` for any UID logged in to it locally or remotely, and `dbus-daemon` run and slowly increase their time running from about 1 second to several seconds, and their collective CPU usage go from 9% to almost 100%. In `btop` it's a noticeable pulsing in the CPU history that gets stronger and longer. Their memory usage also seems to slightly increase over time.

Initially I noticed this when I reconfigured this PC with a new SSD (Crucial brand) and set up a ZFS boot drive following the arch wiki. With this setup, I noticed that after a few days that I couldn't `ssh` into the box due to timeouts, and at its console most things like `sudo` and even `ls /proc` would take a long time. The only way to get back to normal was to reboot the box.

Thinking it might have something to do with the ZFS setup, I created a new setup on a fast USB thumbdrive using btrfs and copied my setup over to it. While this setup didn't slow to a crawl like the ZFS setup did, it did reveal the periodic CPU/process pulsing I described.

Just to see if it had something to do with those filesystems, I created another new setup using another fast USB thumbdrive using ext4 for `/` and `/home` partitions, copied my setup files again to it, and booted. Same thing happened as with the btrfs setup -- no timeouts (yet) but gradual CPU/process and memory creep in `btop`.

Now I'm wondering if it has something to do with my old setup files (something in `/etc` and `/var` maybe?) that I've copied over but I'm not sure where to look/at what to look to verify, and I haven't tried doing a clean install with just the necessary nftables, etc. files.

Has anyone else noticed this on their machine? Any ideas what's going on?

For the record, I haven't seen anything weird pop up in the journal unless it's so minute I've glanced over it -- the bursts don't seem to leave a trace in the logs. And `strace`ing the processes doesn't seem to show anything odd to my eyes. I've also tried both the mainline arch kernel and the lts kernel; lts seems to be slower to develop the issue but still does.

Halp? (also let me know if I've posted this to the wrong forum... sysadmin seemed the best place at first.)

vic_acid

Member

Registered: 2016-12-05

Posts: 13

Re: Periodic high CPU bursts for /sbin/init and systemd --user...

Quick update:

After enabling debug messaging for the kernel and systemd and rebooting, I see a lot of diskseq messages happening, particularly with the built-in SD card reader on the miniPC.

At boot, the kernel or udev assigns the SD card reader a SCSI persistent storage ID in the sd* pattern (e.g. sdb, if I'm booting from sda and don't have any other drives attached to the PC), even if I don't have an SD card in the reader (and I don't). Then it looks like udevd gets triggered every 49-50s (by systemd-udevd, along with the other processes I mentioned previously), causing it to poll the SD card reader and repeatedly increment the diskseq for it and run any udev rules that may apply for it (e.g. /usr/lib/udev/rules.d/60-persistent-storage.rules). My guess is that this is what's causing the CPU pulses, but I'm not sure why they get longer/larger over time yet.

I'm now building out a fresh'n'clean boot USB drive install to try, and I'm only going to copy necessary gateway/router/firewall files to it and see if my wanton copying of all files from the previous drive(s) is somehow botching some systemd or other system config somewhere.

Again, if anyone has experience with similar issues or can enlighten me to maybe what's going on, particularly with diskseq shenanigans, I'd appreciate it.

jeanmarc

Member

Registered: 2019-12-18

Posts: 17

Re: Periodic high CPU bursts for /sbin/init and systemd --user...

Hi,
I experience the same with times /usr/bin/dbus-daemon and /sbin/init eat CPU. I also own a Zotac z-box.
Did you find anything ? l cannot find what the cause is
thanks

vic_acid

Member

Registered: 2016-12-05

Posts: 13

Re: Periodic high CPU bursts for /sbin/init and systemd --user...

Hey, @jeanmarc: I think I did figure something out with this. If I put an SD card in the built-in card reader, it seems to fix the issue. Does your Z-box have a built-in SD card reader on a USB bus like mine does? (I'm using a Nano-C Z-box)

Let me know if that helps at all. I probably need to post an issue on github to let the systemd folks know and see if this is unexpected.

jeanmarc

Member

Registered: 2019-12-18

Posts: 17

Re: Periodic high CPU bursts for /sbin/init and systemd --user...

Hi vic_acid,
Yes i own a CI329 (https://www.zotac.com/us/product/mini_p … -nano#spec)

[jeanmarc@zbox ~]$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 007: ID 8087:0aaa Intel Corp. Bluetooth 9460/9560 Jefferson Peak (JfP)
Bus 001 Device 006: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 001 Device 005: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 001 Device 004: ID 0bda:0153 Realtek Semiconductor Corp. 3-in-1 (SD/SDHC/SDXC) Card Reader
Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 002: ID 2341:8036 Arduino SA Leonardo (CDC ACM, HID)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I am gonna try to put a sdcard and see how it goes. Thanks for the tips

jeanmarc

Member

Registered: 2019-12-18

Posts: 17

Re: Periodic high CPU bursts for /sbin/init and systemd --user...

Hi vic_acid,
So far, putting a SD did the trick

RaumZeit

Member

Registered: 2012-04-04

Posts: 7

Re: Periodic high CPU bursts for /sbin/init and systemd --user...

Hi vic_acid,
So far, putting a SD did the trick

I second that with a Zbox CA621Nano. Inserting an SD card turns the load of the systemd process down from 100% to almost nothing.

Does anyone have an idea how to solve this issue without inserting an SD card?