Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

prMoriarty · 2025-09-18 15:41:13

I use Archlinux x86_64 on my Chromebook inside Crostiny (LXC). During last systemd update
3 core/systemd 257.9-1 -> 258-1
2 core/systemd-libs 257.9-1 -> 258-1
1 core/systemd-sysvcompat 257.9-1 -> 258-1
container crashed or closed during this update procedure and cannot be started again. Lucky me, I have backup just before systemd update, so I have 100% reproduce, but no understanding what exactly happened or how to fix.

Last message in console:
(2/3) upgrading systemd
Creating group 'clock' with GID 966.

then archlinux container was closed and cannot be started again.

seth · 2025-09-18 19:19:59

To be clear, the container crashes *during* the update, not on subsequent boots?

pacman -Syu -v --debug

Does

sudo touch /etc/systemd/do-not-udevadm-trigger-on-update

*before* the update prevent the crash?
https://bugs.archlinux.org/task/77789
https://gitlab.archlinux.org/archlinux/ … 97e8b3baff

CharlesMeng · 2025-09-19 01:06:23

I also got following error when installing from scratch or creating ISO.

Failed to check for chroot() environment: Function not implemented
Failed to connect to system scope bus via local transport: No data available

Error during creating ISO.
(13/15) Reloading system bus configuration...
Failed to check for chroot() environment: Function not implemented
Failed to connect to system scope bus via local transport: No data available

Downgraded to 257.9 and the problem was gone.

Last edited by CharlesMeng (2025-09-19 01:07:50)

seth · 2025-09-19 08:52:30

https://bbs.archlinux.org/viewtopic.php … 3#p2262413

prMoriarty · 2025-09-19 19:22:58

I tried new systemd build 258-2 and still the same behaviour.
I also tried to make new fresh container and saw a little more detailed error

Creating group 'clock' with GID 968.
systemd 258-2-arch running in system mode (+PAM +AUDIT -SELINUX +APPARMOR -IMA +IPE +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP -SYSVINIT +LIBARCHIVE)
Detected virtualization lxc.
Detected architecture x86-64.
Found tmpfs on /sys/fs/cgroup/, assuming legacy hierarchy.
Detected cgroup v1 hierarchy at /sys/fs/cgroup/, which is no longer supported by current version of systemd.
Please instruct your initrd to mount cgroup v2 (unified) hierarchy,
possibly by removing any stale kernel command line options, such as:
systemd.legacy_systemd_cgroup_controller=1
systemd.unified_cgroup_hierarchy=0
[!!!!!!] Detected unsupported legacy cgroup hierarchy, refusing execution.
Exiting PID 1...
Error: write /dev/pts/ptmx: file already closed

seth · 2025-09-19 19:43:35

https://bbs.archlinux.org/viewtopic.php … 3#p2262413 is only about "Failed to check for chroot() environment: Function not implemented", that should now be gone?
Wrt the container restart, have you tried to touch /etc/systemd/do-not-udevadm-trigger-on-update ?

prMoriarty · 2025-09-19 19:45:03

Results of

pacman -Syu -v --debug

https://paste.c-net.org/LaineyDrafted

tried

sudo touch /etc/systemd/do-not-udevadm-trigger-on-update

no difference

prMoriarty · 2025-09-19 20:41:33

I run lxc with debug logs too

  lxc --debug exec arch -- bash

but before container was force closed, I saw only one extra message from lxc

 DEBUG  [2025-09-19T20:57:50+01:00] WebsocketRecvStream got message barrier

seth · 2025-09-19 20:51:02

Does just "groupadd -g 969 clock" cause the same then?
Edit: trigger seems https://gitlab.archlinux.org/archlinux/ … type=heads ?

Last edited by seth (2025-09-19 20:53:45)

prMoriarty · 2025-09-19 21:26:48

seth wrote:

Does just "groupadd -g 969 clock" cause the same then?

I tried, no it does not cause container stop

seth · 2025-09-19 21:53:42

Does "/usr/share/libalpm/scripts/systemd-hook sysusers"resp. just /usr/bin/systemd-sysusers ?
The next hook would probably run "/usr/share/libalpm/scripts/systemd-hook daemon-reload-system" and that's actually way more likely to cause trouble…

prMoriarty · 2025-09-19 22:08:02

Both command does not provide responds and does not cause same issue

[root@archtest ~]# /usr/share/libalpm/scripts/systemd-hook sysusers
[root@archtest ~]# /usr/share/libalpm/scripts/systemd-hook daemon-reload-system

Edit: but important that both was NOT run in systemd 258 environment, as I cannot update to systemd 258

Last edited by prMoriarty (2025-09-19 22:10:18)

seth · 2025-09-21 08:10:52

Maybe yo also should not… https://bbs.archlinux.org/viewtopic.php?id=308380

Add "NoExtract = usr/share/libalpm/scripts/systemd-hook" to yoru pacman.conf, then edit usr/share/libalpm/scripts/systemd-hook

…
  catalog)
    /usr/bin/journalctl --update-catalog
    ;;
  daemon-reload-system)
    echo "daemon-reload-system" # new line
    # systemd_live # comment this
    # /usr/bin/systemctl --system daemon-reload  # comment this
    ;;
  daemon-reload-user)
    echo "daemon-reload-user" # new line
    # systemd_live # comment this
    # /usr/bin/systemctl reload 'user@*.service' # comment this
    ;;
  hwdb)
    /usr/bin/systemd-hwdb --usr update
    ;;
  sysctl)
    systemd_live
    /usr/lib/systemd/systemd-sysctl
    ;;
  sysusers)
    echo "sysusers" # new line
    # /usr/bin/systemd-sysusers # comment this
    ;;
  tmpfiles)
    /usr/bin/systemd-tmpfiles --create
    ;;

and see whether you can udpate systemdumb.
If yes, check what's triggered in which order and then undo the script changes and attempt whether either

/usr/share/libalpm/scripts/systemd-hook sysusers
/usr/share/libalpm/scripts/systemd-hook daemon-reload-system

crashes the system.

prMoriarty · 2025-09-21 19:53:14

No difference, the same console output https://paste.c-net.org/ListenerLimos

Just to double-check here is my edited pacman.conf https://paste.c-net.org/SouvenirHangout

and my edited systemd-hook https://paste.c-net.org/PeopleDeath

I also run with debug logs

 pacman -Syu -v --debug

https://paste.c-net.org/RushesTruck

in debug logs I can see that settings worked
debug: usr/share/libalpm/scripts/systemd-hook is in NoExtract, skipping extraction of /usr/share/libalpm/scripts/systemd-hook

seth · 2025-09-21 20:01:45

**grrrr***

Ok, put "set -x" at the top of /usr/share/libalpm/scripts/systemd-hook (right below the shebang) and "sleep 10" at the bottom (right above "exit 0") which shall
1. make it print what it does
2. wait 10s at the end to signal that this has worked and isn't the cause of the crash

prMoriarty · 2025-09-21 20:27:47

hm... I tried 2 times, carefully checked that I edited systemd-hook correctly. NO DIFFERENCE. The same lines before crash without any delays
:: Processing package changes...
(1/3) upgrading systemd-libs
(2/3) upgrading systemd
Creating group 'clock' with GID 969.

seth · 2025-09-21 20:54:32

If you add

echo $*
sleep 10

to the top (hoping that the first invocation is bogus and kills everything we at least get to see it here)

The "Creating group 'clock' with GID 969." call there has to come from systemd-sysusers, so add "NoExtract = usr/bin/systemd-sysusers" and move /usr/bin/systemd-sysusers to /usr/bin/not.systemd-sysusers and maybe symlink /bin/true to /usr/bin/systemd-sysusers …
Although the invocation could not come from the /usr/share/libalpm/scripts/systemd-hook

prMoriarty · 2025-09-21 22:41:38

I added

 echo $*
sleep 10

to /usr/share/libalpm/scripts/systemd-hook below shebang but did not notice delay before crash

looks like symlink /bin/true does not acceptable but, at least I saw new output before crash

(1/3) upgrading systemd-libs
(2/3) upgrading systemd
/tmp/alpm_DLJ6Ag/.INSTALL: line 15: systemd-sysusers: command not found

Scimmia · 2025-09-21 22:44:51

How is that possible? What does `pacman -Qikk systemd` show?

Edit, nevermind, I'd missed where this thread had gone.

Last edited by Scimmia (2025-09-21 22:46:36)

seth · 2025-09-22 08:35:14

Yeah, we're breaking the system left and right to figure what's actually crashing the container - and apparently it's not sysusers.

Pacman is probably very smart and runs the hook from the package, not the system.

seth wrote:

The next hook would probably run "/usr/share/libalpm/scripts/systemd-hook daemon-reload-system" and that's actually way more likely to cause trouble…

NoExtract = usr/bin/systemd-detect-virt

mv /usr/bin/systemd-detect-virt /usr/bin/not.systemd-detect-virt
ln -s /usr/bin/true /usr/bin/systemd-detect-virt

This shall fake a chroot environment and make the hook abort instead of reloading the dameon.
Then reload the daemon manually… and whatch your container die?

prMoriarty · 2025-09-22 12:28:18

The same result, crash.

Also I checked pacman debug logs.
> debug: usr/bin/systemd-detect-virt is in NoExtract, skipping extraction of /usr/bin/systemd-detect-virt

> debug: executing "/usr/bin/bash" under chroot "/"

this is only one mention of chroot that I noticed in logs

before replacing systemd-detect-virt I also cheched what is actual result it provides

[root@archtest ~]# /usr/bin/systemd-detect-virt
lxc

seth · 2025-09-22 13:59:20

/usr/bin/systemd-detect-virt --chroot && echo "is chroot"

NoExtract = usr/bin/systemctl

mv /usr/bin/systemctl /usr/bin/systemctl.bin
printf '#!/bin/sh\necho $*\nsleep 10\nexec /usr/bin/systemctl.bin "$@"\n' > /usr/bin/systemctl
chmod ugo+x /usr/bin/systemctl

prMoriarty · 2025-09-22 14:18:36

[root@archtest ~]# /usr/bin/systemd-detect-virt --chroot && echo "is chroot"
[root@archtest ~]# /usr/bin/systemd-detect-virt --chroot
[root@archtest ~]# echo "is chroot"
is chroot

pacman finished successfully, container still running

:: Processing package changes...
(1/3) upgrading systemd-libs                                                                                                     [##############################################################################] 100%
(2/3) upgrading systemd                                                                                                          [##############################################################################] 100%
Creating group 'clock' with GID 969.
/tmp/alpm_uoUlf3/.INSTALL: line 44: systemctl: command not found
/tmp/alpm_uoUlf3/.INSTALL: line 47: systemctl: command not found
/tmp/alpm_uoUlf3/.INSTALL: line 8: systemctl: command not found
:: This is a systemd feature update. You may want to have a look at
   NEWS for what changed, or if you observe unexpected behavior:
     /usr/share/doc/systemd/NEWS
New optional dependencies for systemd
    apparmor: additional security features
(3/3) upgrading systemd-sysvcompat                                                                                               [##############################################################################] 100%
:: Running post-transaction hooks...
( 1/11) Creating system user accounts...
( 2/11) Updating journal message catalog...
( 3/11) Reloading system manager configuration...
  Skipped: Current root is not booted.
( 4/11) Reloading user manager configuration...
  Skipped: Current root is not booted.
( 5/11) Updating udev hardware database...
( 6/11) Restarting marked services...
  Skipped: Current root is not booted.
( 7/11) Applying kernel sysctl settings...
  Skipped: Current root is not booted.
( 8/11) Creating temporary files...
( 9/11) Reloading device manager configuration...
  Skipped: Current root is not booted.
(10/11) Arming ConditionNeedsUpdate...
(11/11) Reloading system bus configuration...
  Skipped: Current root is not booted.

Edit: I am not sure why it was not found. I executed comands that you provided and checked new file

[root@archtest ~]# printf '#!/bin/sh\necho $*\nsleep 10\nexec /usr/bin/systemctl.bin "$@"\n' > /usr/bin/systemctl
[root@archtest ~]# cat /usr/bin/systemctl
#!/bin/sh
echo $*
sleep 10
exec /usr/bin/systemctl.bin "$@"
[root@archtest ~]# chmod ugo+x /usr/bin/systemctl

Last edited by prMoriarty (2025-09-22 14:25:36)

seth · 2025-09-22 14:27:27

/tmp/alpm_uoUlf3/.INSTALL: line 44: systemctl: command not found
/tmp/alpm_uoUlf3/.INSTALL: line 47: systemctl: command not found
/tmp/alpm_uoUlf3/.INSTALL: line 8: systemctl: command not found

You didn't create or +x the systemctl wrapper script, but it's

/usr/bin/systemctl reload 'user@*.service'

systemctl --user list-units

And then see whether

/usr/bin/systemctl reload 'user@*.service'

still crashes it…

prMoriarty · 2025-09-22 14:47:32

/usr/bin/systemctl was absent on filesystem, probably it was removed in process of pacman execution. Looks like NoExtract = usr/bin/systemctl prevented just extracting new one, but not deletion old one.

I extracted systemctl from systemd 258-2 package and executed requested commands. My test environment is minimal just created lxd container, so I did not create users or any extra tools and commands were run just from root. No crash.

[root@archtest ~]# /usr/bin/systemctl reload 'user@*.service'
reload user@*.service
Warning: The unit file, source configuration file or drop-ins of user@1000.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for user@1000.service failed because the control process exited with error code.
See "systemctl status user@1000.service" and "journalctl -xeu user@1000.service" for details.
[root@archtest ~]# systemctl --user list-units
--user list-units
Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)
[root@archtest ~]# /usr/bin/systemctl reload 'user@*.service'
reload user@*.service
Warning: The unit file, source configuration file or drop-ins of user@1000.service changed on disk. Run 'systemctl daemon-reload' to reload units.
user@1000.service is not active, cannot reload.
[root@archtest ~]# systemctl daemon-reload
daemon-reload
[root@archtest ~]# /usr/bin/systemctl reload 'user@*.service'
reload user@*.service
user@1000.service is not active, cannot reload.
[root@archtest ~]# systemctl --user list-units
--user list-units
Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)
[root@archtest ~]# /usr/bin/systemctl reload 'user@*.service'
reload user@*.service
user@1000.service is not active, cannot reload.
[root@archtest ~]#

Last edited by prMoriarty (2025-09-22 14:48:20)

Arch Linux

#1 2025-09-18 15:41:13

Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#2 2025-09-18 19:19:59

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#3 2025-09-19 01:06:23

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#4 2025-09-19 08:52:30

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#5 2025-09-19 19:22:58

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#6 2025-09-19 19:43:35

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#7 2025-09-19 19:45:03

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#8 2025-09-19 20:41:33

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#9 2025-09-19 20:51:02

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#10 2025-09-19 21:26:48

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#11 2025-09-19 21:53:42

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#12 2025-09-19 22:08:02

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#13 2025-09-21 08:10:52

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#14 2025-09-21 19:53:14

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#15 2025-09-21 20:01:45

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#16 2025-09-21 20:27:47

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#17 2025-09-21 20:54:32

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#18 2025-09-21 22:41:38

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#19 2025-09-21 22:44:51

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#20 2025-09-22 08:35:14

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#21 2025-09-22 12:28:18

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#22 2025-09-22 13:59:20

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#23 2025-09-22 14:18:36

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#24 2025-09-22 14:27:27

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

#25 2025-09-22 14:47:32

Re: Archlinux stops working inside Crostiny (ChromeOS) after systemd 258

Board footer