You are not logged in.

#1 2022-01-08 16:08:24

mattange
Member
Registered: 2020-04-30
Posts: 6

"systemctl status" essentially fails after upgrade

Hi All,

looking for some guidance to solve a bizarre (for me...) error...

I have a physical machine running Arch, in turn running some KVM based virtual machines also with Arch. Everything works as expected after recent upgrades, bar - as far as i can see so far - the "systemctl status" command.
Network works, kvms are up and running and "systemctl status" in the machines works as expected.

On the physical machine (and i am not even sure this is relevant) when trying to check system status with "systemctl status" (even as sudoer) I get this:

16:05 [username@machinename : ~]$ systemctl status
Failed to dump process list for 'machinename', ignoring: Operation not supported
● machinename
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2022-01-08 15:59:14 GMT; 9min ago
   CGroup: /
16:05 [username@machinename : ~]$

So there is an error about failure to dump process list, but pstree shows everything seems to be working just fine? What config may have changed that broke this (i recently updated to Systemd-250... maybe that's the reason?)
for all intent and purposes, the machine seems to be working fine... and so appear to do the KVMs...

Can anybody shed some light on how i can get my usual systemctl tree back? any error that i should search for anywhere?
Notably, "systemctl status [servicename]" works as expected showing the status and operation of the relevant service.

Any help appreciated, thanks in advance!

Offline

#2 2022-01-08 19:13:36

Head_on_a_Stick
Member
From: London
Registered: 2014-02-20
Posts: 7,680
Website

Re: "systemctl status" essentially fails after upgrade

mattange wrote:

i recently updated to Systemd-250... maybe that's the reason?

So does downgrading the systemd package fix things?

Can we see

type systemctl
stat /usr/bin/systemctl

Offline

#3 2022-01-10 09:30:35

mattange
Member
Registered: 2020-04-30
Posts: 6

Re: "systemctl status" essentially fails after upgrade

Hi - thanks for prompt response.
To answer the questions:

So does downgrading the systemd package fix things? --> yes, just tried to downgrade only systemd to 249 and i get totally fine info on "systemctl status".
To clarify, this works... i get regular info with "systemctl status" after this is run...

sudo pacman -U /var/cache/pacman/pkg/systemd-249.7-2-x86_64.pkg.tar.zst

What you asked from v250 produces the below (i.e. run before the downgrade):

08:48 [username@machinename : ~]$ type systemctl
systemctl is /usr/bin/systemctl
08:48 [username@machinename : ~]$ stat /usr/bin/systemctl
  File: /usr/bin/systemctl
  Size: 272664          Blocks: 536        IO Block: 4096   regular file
Device: 254,4   Inode: 278991      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-01-10 00:15:00.868050067 +0000
Modify: 2022-01-05 08:01:36.000000000 +0000
Change: 2022-01-08 15:46:36.863787644 +0000
Birth: 2022-01-08 15:46:36.863787644 +0000

I can't see anything wrong here but don't really know what to look for!

thanks

regards

Offline

#4 2022-01-10 14:52:14

robson75
Member
From: Poland
Registered: 2020-06-28
Posts: 144

Re: "systemctl status" essentially fails after upgrade

On my systemd 250 version the systemctl status command works fine

● pc
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2022-01-10 15:39:59 CET; 6min ago
   CGroup: /
           ├─init.scope
           │ └─1 /sbin/init
           ├─system.slice
           │ ├─ModemManager.service
           │ │ └─362 /usr/bin/ModemManager
           │ ├─NetworkManager.service
           │ │ └─340 /usr/bin/NetworkManager --no-daemon
           │ ├─accounts-daemon.service
           │ │ └─317 /usr/lib/accounts-daemon
           │ ├─acpid.service
           │ │ └─318 /usr/bin/acpid --foreground --netlink
           │ ├─avahi-daemon.service
           │ │ ├─320 "avahi-daemon: running [pc.local]"
           │ │ └─332 "avahi-daemon: chroot helper"
           │ ├─avahi-dnsconfd.service
           │ │ └─321 /usr/bin/avahi-dnsconfd -s
           │ ├─cronie.service
           │ │ └─324 /usr/bin/crond -n
           │ ├─dbus-broker.service
           │ │ ├─315 /usr/bin/dbus-broker-launch --scope system --audit
           │ │ └─316 dbus-broker --log 4 --controller 9 --machine-id 5c007ac3d6d54454bc09920cc675de84 --max-bytes 536870912 --max-fds 4096 --max-matches 131072 --audit
           │ ├─hddtemp.service
           │ │ └─325 /usr/bin/hddtemp -dF /dev/sda
           │ ├─lightdm.service
           │ │ ├─359 /usr/bin/lightdm
           │ │ └─374 /usr/lib/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
           │ ├─polkit.service
           │ │ └─328 /usr/lib/polkit-1/polkitd --no-debug
           │ ├─rdisc.service
           │ │ └─355 /usr/bin/rdisc -f -t
           │ ├─rtkit-daemon.service
           │ │ └─552 /usr/lib/rtkit-daemon
           │ ├─sshd.service
           │ │ └─356 "sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups"
           │ ├─systemd-journald.service
           │ │ └─202 /usr/lib/systemd/systemd-journald
           │ ├─systemd-logind.service
           │ │ └─329 /usr/lib/systemd/systemd-logind
           │ ├─systemd-resolved.service
           │ │ └─309 /usr/lib/systemd/systemd-resolved
           │ ├─systemd-timesyncd.service
           │ │ └─310 /usr/lib/systemd/systemd-timesyncd
           │ ├─systemd-udevd.service
           │ │ └─220 /usr/lib/systemd/systemd-udevd
           │ ├─udisks2.service
           │ │ └─799 /usr/lib/udisks2/udisksd
           │ ├─upower.service
           │ │ └─710 /usr/lib/upowerd
           │ ├─wpa_supplicant.service
           │ │ └─330 /usr/bin/wpa_supplicant -u -s -O /run/wpa_supplicant
           │ └─xinetd.service
           │   └─358 /usr/bin/xinetd -dontfork
           └─user.slice
             └─user-1000.slice
               ├─session-1.scope
               │ ├─ 382 lightdm --session-child 12 15
               │ ├─ 409 /usr/bin/gnome-keyring-daemon --daemonize --login
               │ ├─ 419 xfce4-session
               │ ├─ 533 xfsettingsd
               │ ├─ 540 xfce4-panel
               │ ├─ 546 Thunar --daemon
               │ ├─ 551 xfdesktop
               │ ├─ 557 /usr/lib/xfce4/panel/wrapper-2.0 /usr/lib/xfce4/panel/plugins/libwhiskermenu.so 1 12582919 whiskermenu "Menu Whisker" "Wyświetla aktywatory zainstalowanych programów rozmieszczone w menu"
               │ ├─ 565 /usr/lib/xfce4/panel/wrapper-2.0 /usr/lib/xfce4/panel/plugins/libxfce4-sensors-plugin.so 5 12582932 xfce4-sensors-plugin "Wtyczka czujników" "Wyświetla wskazania czujników sprzętowych"
               │ ├─ 566 /usr/lib/xfce4/panel/wrapper-2.0 /usr/lib/xfce4/panel/plugins/libsystray.so 2 12582933 systray "Wtyczka zasobnika stanu" "Zapewnia elementy powiadamiające o stanie (wskaźniki aplikacji) i starsze elementy zasobnika systemowego"
               │ ├─ 595 pa-applet
               │ ├─ 597 python3 /usr/bin/redshift-gtk
               │ ├─ 609 compiz --replace ccp
               │ ├─ 610 emerald --replace
               │ ├─ 611 conky -c /home/robson/.conky/conkyrc_info
               │ ├─ 612 conky -c /home/robson/.conky/mail_info
               │ ├─ 614 /usr/lib/polkit-gnome/polkit-gnome-authentication-agent-1
               │ ├─ 620 nm-applet
               │ ├─ 624 xfce4-power-manager
               │ ├─ 629 /usr/lib/xfce4/notifyd/xfce4-notifyd
               │ ├─ 679 clipit
               │ ├─ 700 /usr/lib/geoclue-2.0/demos/agent
               │ ├─ 828 /usr/bin/redshift -v
               │ ├─3223 xfce4-terminal
               │ ├─3240 zsh
               │ └─3391 systemctl status
               └─user@1000.service
                 ├─app.slice
                 │ ├─app-dbus\x2d:1.9\x2dorg.a11y.atspi.Registry.slice
                 │ │ └─dbus-:1.9-org.a11y.atspi.Registry@0.service
                 │ │   └─517 /usr/lib/at-spi2-registryd --use-gnome-session
                 │ ├─at-spi-dbus-bus.service
                 │ │ ├─504 /usr/lib/at-spi-bus-launcher
                 │ │ ├─509 /usr/bin/dbus-broker-launch --config-file=/usr/share/defaults/at-spi2/accessibility.conf --scope user
                 │ │ └─510 dbus-broker --log 4 --controller 9 --machine-id 5c007ac3d6d54454bc09920cc675de84 --max-bytes 100000000000000 --max-fds 6400000 --max-matches 5000000000
                 │ ├─dconf.service
                 │ │ └─922 /usr/lib/dconf-service
                 │ ├─gvfs-daemon.service
                 │ │ ├─491 /usr/lib/gvfsd
                 │ │ ├─496 /usr/lib/gvfsd-fuse /run/user/1000/gvfs -f
                 │ │ └─901 /usr/lib/gvfsd-trash --spawner :1.5 /org/gtk/gvfs/exec_spaw/0
                 │ ├─gvfs-metadata.service
                 │ │ └─919 /usr/lib/gvfsd-metadata
                 │ └─gvfs-udisks2-volume-monitor.service
                 │   └─796 /usr/lib/gvfs-udisks2-volume-monitor
                 ├─init.scope
                 │ ├─390 /usr/lib/systemd/systemd --user
                 │ └─394 "(sd-pam)"
                 └─session.slice
                   ├─dbus.service
                   │ ├─ 425 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
                   │ ├─ 512 /usr/lib/xfce4/xfconf/xfconfd
                   │ └─1363 /usr/lib/gconfd-2
                   └─pulseaudio.service
                     ├─538 /usr/bin/pulseaudio --daemonize=no --log-target=journal
                     └─751 /usr/lib/pulse/gsettings-helper

I think you have something wrong configured.


Arch Linux Xfce - 64Bit Linux User #621110

Offline

#5 2022-01-10 14:59:38

mattange
Member
Registered: 2020-04-30
Posts: 6

Re: "systemctl status" essentially fails after upgrade

Thanks @robson75
I am sure you're right, but I presume that's what i am also trying to ascertain, as downgrading the package works just fine and parts of 250 clearly do too.
Let's see if @Head_on_a_Stick has some further thoughts (or anyone else for that matter!)

Kind regards

Offline

#6 2022-01-10 15:21:50

robson75
Member
From: Poland
Registered: 2020-06-28
Posts: 144

Re: "systemctl status" essentially fails after upgrade

And have you followed the tips given in this thread?
https://bbs.archlinux.org/viewtopic.php … 6#p2012166


Arch Linux Xfce - 64Bit Linux User #621110

Offline

#7 2022-01-10 16:30:41

seth
Member
Registered: 2012-09-03
Posts: 49,982

Re: "systemctl status" essentially fails after upgrade

I can't imagine that it's the oom score.
@mattange, are you willing to disclose the "machinename" (assuming that's not literally it)

Offline

#8 2022-01-10 20:41:04

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: "systemctl status" essentially fails after upgrade

Is the result the same if you shutdown all virtual machines and containers?

The error message looks to be produced by https://github.com/systemd/systemd/blob … how.c#L741
Edit:
Also is the system using a custom kernel?  Does the system have any customization of cgroups?

Last edited by loqs (2022-01-10 20:50:58)

Offline

#9 2022-01-11 23:50:55

alex19EP
Package Maintainer (PM)
From: Russia, Moscow
Registered: 2021-07-05
Posts: 2
Website

Re: "systemctl status" essentially fails after upgrade

loqs wrote:

Is the result the same if you shutdown all virtual machines and containers?
Also is the system using a custom kernel?  Does the system have any customization of cgroups?

indeed for me this bug appears if win11 is running in libvirt virtual machine.

Offline

#10 2022-01-12 01:47:43

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: "systemctl status" essentially fails after upgrade

Any additional output from:

SYSTEMD_LOG_LEVEL=7 SYSTEMD_PAGER= systemctl status

Edit:
Does systemd-cgls work as expected?

Last edited by loqs (2022-01-12 01:54:24)

Offline

#11 2022-01-12 02:07:34

Toolybird
Member
Registered: 2017-09-30
Posts: 72

Re: "systemctl status" essentially fails after upgrade

alex19EP wrote:

indeed for me this bug appears if win11 is running in libvirt virtual machine.

Not just Win11, even archiso in a libvirt VM seems to trigger it. Weird.

Either libvirt is doing something funky with cgroups or it's a systemd regression.

Offline

#12 2022-01-12 05:13:40

Toolybird
Member
Registered: 2017-09-30
Posts: 72

Re: "systemctl status" essentially fails after upgrade

Offline

#13 2022-01-12 08:15:26

mattange
Member
Registered: 2020-04-30
Posts: 6

Re: "systemctl status" essentially fails after upgrade

Hi All
just saw all replies in one. I notice there was a bug report raised on SystemD, thanks for that. Will monitor.

Responding to individual questions where i can:
1) Is the result the same if you shutdown all virtual machines and containers? --> not tried yet, but from follow up posts it seems to be the culprit. Notably, all is fine INSIDE the VM
2) machine name disclosed below (no particular issues there)
3) SYSTEMD_LOG_LEVEL=7 SYSTEMD_PAGER= systemctl status output:

running_in_chroot(): Permission denied
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/dbus/system_bus_socket...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → HELLO
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.21 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Bus n/a: changing state HELLO → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=:1.18 destination=:1.21 path=n/a interface=n/a member=n/a cookie=1778 reply_cookie=2 signature=a{sv} error-name=n/a error-message=n/a
● brainy
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2022-01-12 06:15:28 GMT; 1h 56min ago
   CGroup: /
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=:1.18 destination=:1.21 path=n/a interface=n/a member=n/a cookie=1779 reply_cookie=3 signature=s error-name=org.freedesktop.DBus.Error.NotSupported error-message=Operation not supported
Failed to dump process list for 'brainy', ignoring: Operation not supported
Bus n/a: changing state RUNNING → CLOSED

4) Does systemd-cgls work as expected? --> actually did not know this command but ... YES this shows all processes as i would expect.
5) Also is the system using a custom kernel?  --> no
6) Does the system have any customization of cgroups? --> i don't think so, don't remember having added anything here (but happy to be reminded how to look into this!)
6) issue still there on 250.2-2

Thanks all for looking

Last edited by mattange (2022-01-12 08:19:12)

Offline

#14 2022-01-12 10:54:02

alex19EP
Package Maintainer (PM)
From: Russia, Moscow
Registered: 2021-07-05
Posts: 2
Website

Re: "systemctl status" essentially fails after upgrade

loqs wrote:

Any additional output from:

SYSTEMD_LOG_LEVEL=7 SYSTEMD_PAGER= systemctl status

Edit:
Does systemd-cgls work as expected?

running_in_chroot(): Permission denied
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/dbus/system_bus_socket...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → HELLO
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.152 path=n/a interface=n/a member=n/a cookie=4294967295 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Bus n/a: changing state HELLO → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=:1.1 destination=:1.152 path=n/a interface=n/a member=n/a cookie=1946 reply_cookie=2 signature=a{sv} error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=:1.1 destination=:1.152 path=n/a interface=n/a member=n/a cookie=1947 reply_cookie=3 signature=s error-name=org.freedesktop.DBus.Error.NotSupported error-message=Operation not supported
Failed to dump process list for 'alex-pc', ignoring: Operation not supported
Bus n/a: changing state RUNNING → CLOSED
● alex-pc
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2022-01-12 12:38:01 MSK; 1h 11min ago
   CGroup: /

systemd-cgls works fine.

Offline

#15 2022-01-12 18:35:05

carpenike
Member
Registered: 2022-01-12
Posts: 1

Re: "systemctl status" essentially fails after upgrade

Also seeing this same thing. Also running libvirtd with a windows VM.


Failed to dump process list for 'rydesk', ignoring: Operation not supported
● rydesk
    State: degraded
     Jobs: 0 queued
   Failed: 4 units
    Since: Wed 2022-01-12 09:04:30 EST; 4h 29min ago
   CGroup: /

[root@rydesk ~]# SYSTEMD_LOG_LEVEL=7 SYSTEMD_PAGER= systemctl status
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/systemd/private...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=1 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=a{sv} error-name=n/a error-message=n/a
● rydesk
    State: degraded
     Jobs: 0 queued
   Failed: 4 units
    Since: Wed 2022-01-12 09:04:30 EST; 4h 31min ago
   CGroup: /
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=2 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.NotSupported error-message=Operation not supported
Failed to dump process list for 'rydesk', ignoring: Operation not supported
Bus n/a: changing state RUNNING → CLOSED

Last edited by carpenike (2022-01-12 18:35:59)

Offline

#16 2022-01-12 18:52:35

dogknowsnx
Member
Registered: 2021-04-12
Posts: 648

Re: "systemctl status" essentially fails after upgrade

carpenike wrote:

Also seeing this same thing. Also running libvirtd with a windows VM.

See post #12


Notifications for Arch Linux package updates
RI - Rest your Eyes and Self

"We are eternal, all this pain is an illusion" - Maynard James Keenan

Offline

Board footer

Powered by FluxBB