after update system hangs for 90sec and needs extra ^d key ...

tronics · 2022-12-02 10:30:51

Hallo everybody,

first of all: Arch Linux is a great system!

My problem:

Since some days, just after a complete update, the system hangs for 90 seconds
and needs an extra key ^D to start over to the login GUI...

I already tries several "bug fixes"/"issues", but nothing helped.

So, I just have only a real screen shot... but I do not know, how to add it here.
So please help!

If you have any idea, please send a response.

Thank you in advance!

regards,
Werner

seth · 2022-12-02 10:40:59

90s is the systemd service timeout - instead of sending any pictures, post the system journal

sudo journalctl -b | curl -F 'file=@-' 0x0.st

tronics · 2022-12-02 11:43:15

Thank you for your reply. Just learned something new... :-)

seth wrote:

90s is the systemd service timeout - instead of sending any pictures, post the system journal
sudo journalctl -b | curl -F 'file=@-' 0x0.st

Here is my sudo journalctl -b @ 0x0.st.

Thanx again!

Trilby · 2022-12-02 14:25:39

You neglected to mention being dropped into the emergency shell on boot. It seems the filesystem(s) fail to be detected ... at least the first time. This has nothing to do with starting the GUI - this is much much earlier than that.

Can you confirm you are dropped to the emergency shell? And that's where you hit Ctrl-D to exit? And it just "magically" boots successfully after this?

I'm not sure how / why it'd work the second time but not the first if no changes were made from within the emergency shell (aside perhaps from some odd race-like conditioning in the hardware initialization ... which I've never heard of in this context). But as a quick test, I'd regenerate the initramfs (and bootmanger config if you use GRUB), then see if it boots properly.

Last edited by Trilby (2022-12-02 14:26:37)

seth · 2022-12-02 16:53:20

Dez 02 10:32:39 werners-E756 systemd[1]: Started Emergency Shell.
…
Dez 02 10:32:39 werners-E756 @ystemctl[1070]: Failed to switch root: Could not resolve init executable /usr/lib/systemd/systemd: No such file or directory
Dez 02 10:32:39 werners-E756 systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
Dez 02 10:32:39 werners-E756 systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
Dez 02 10:32:39 werners-E756 systemd[1]: Failed to start Switch Root.
Dez 02 10:32:39 werners-E756 systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.

This looks *a lot* like https://bbs.archlinux.org/viewtopic.php?id=281732 though

Dez 02 10:32:39 werners-E756 systemd[1]: dev-disk-by\x2dlabel-ROOT.device: Job dev-disk-by\x2dlabel-ROOT.device/start timed out.
Dez 02 10:32:39 werners-E756 systemd[1]: Timed out waiting for device /dev/disk/by-label/ROOT.

systemd seems to try to use the root partition by label - same thing, though:

lsblk -f

cfr · 2022-12-03 05:10:12

Possibly related?

https://access.redhat.com/solutions/4973191 (but seems rather old version and date-wise)
https://forums.centos.org/viewtopic.php?t=74434 (even older and also points to RH)

Last edited by cfr (2022-12-03 05:10:41)

tronics · 2022-12-06 11:26:15

Trilby wrote:

You neglected to mention being dropped into the emergency shell on boot. It seems the filesystem(s) fail to be detected ... at least the first time. This has nothing to do with starting the GUI - this is much much earlier than that.
Can you confirm you are dropped to the emergency shell? And that's where you hit Ctrl-D to exit? And it just "magically" boots successfully after this?

YES, that's true...

Trilby wrote:

I'm not sure how / why it'd work the second time but not the first if no changes were made from within the emergency shell (aside perhaps from some odd race-like conditioning in the hardware initialization ... which I've never heard of in this context). But as a quick test, I'd regenerate the initramfs (and bootmanger config if you use GRUB), then see if it boots properly.

I tried both:

# mkinitcpio -P

and

# bootctl update

but with no effect...

Thank you again for your help
Werner

tronics · 2022-12-06 11:34:11

seth wrote:

[...]
This looks *a lot* like https://bbs.archlinux.org/viewtopic.php?id=281732 though

Hmm, i'm not sure about this item, because I do not use encrypted file systems.

seth wrote:

Dez 02 10:32:39 werners-E756 systemd[1]: dev-disk-by\x2dlabel-ROOT.device: Job dev-disk-by\x2dlabel-ROOT.device/start timed out.
Dez 02 10:32:39 werners-E756 systemd[1]: Timed out waiting for device /dev/disk/by-label/ROOT.

systemd seems to try to use the root partition by label - same thing, though:

lsblk -f

Here is what

lsblk -f

shows:

NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                           
├─sda1 vfat   FAT32 BOOT  E0EA-BFF6                             205,8M    31% /boot
├─sda2 ext4   1.0   ROOT  de0eff4f-9c54-48cc-8a32-644ae55d2f47  648,1G    23% /sysroot
│                                                                             /
└─sda3 swap   1           324269e3-e5d1-4f31-afbb-97201a0ec2e5                [SWAP]
sr0

BTW: To demonstrate, what

systemctl status initrd-switch-root.service

delivers:

× initrd-switch-root.service - Switch Root
     Loaded: loaded (/usr/lib/systemd/system/initrd-switch-root.service; static)
     Active: failed (Result: exit-code) since Tue 2022-12-06 11:24:30 CET; 40min ago
    Process: 597 ExecStart=systemctl --no-block switch-root /sysroot (code=exited, status=1/FAILURE)
   Main PID: 597 (code=exited, status=1/FAILURE)
        CPU: 5ms

Dez 06 11:24:30 werners-E756 systemd[1]: Starting Switch Root...
Dez 06 11:24:30 werners-E756 @ystemctl[597]: Failed to switch root: Could not resolve init executable /usr/lib/systemd/systemd: No such file or di>
Dez 06 11:24:30 werners-E756 systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
Dez 06 11:24:30 werners-E756 systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
Dez 06 11:24:30 werners-E756 systemd[1]: Failed to start Switch Root.
Dez 06 11:24:30 werners-E756 systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.

and

ls -l /usr/lib/systemd/systemd

shows this:

.rwxr-xr-x root root 90 KB Fri Dec  2 22:58:37 2022  /usr/lib/systemd/systemd

Any other ideas?

Thanx and regards
Werner

tronics · 2022-12-06 11:46:24

cfr wrote:

Possibly related?
https://access.redhat.com/solutions/4973191 (but seems rather old version and date-wise)
https://forums.centos.org/viewtopic.php?t=74434 (even older and also points to RH)

1. link: This I did not try, because I have no experience with dracut... and didn't dare using it yet...
2. link: Helps... So it still stays for 90 seconds, but does not stop...

Thanx & regards
Werner

seth · 2022-12-06 15:02:40

Try to select the root partition by UUID

root=UUID=de0eff4f-9c54-48cc-8a32-644ae55d2f47

and also please post your /etc/fstab

tronics · 2022-12-08 07:09:48

seth wrote:

Try to select the root partition by UUID
root=UUID=de0eff4f-9c54-48cc-8a32-644ae55d2f47
and also please post your /etc/fstab

Hmm... How may I change the search to search-by-UUID?

Here my /etc/fstab:

# /dev/sda2 LABEL=ROOT
UUID=de0eff4f-9c54-48cc-8a32-644ae55d2f47       /               ext4            rw,relatime     0 1

# /dev/sda1 LABEL=BOOT
UUID=E0EA-BFF6          /boot           vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro      0 2

# /dev/sda3 LABEL=SWAP
UUID=324269e3-e5d1-4f31-afbb-97201a0ec2e5       none            swap            defaults        0 0

Maybe this helps too:

$ sudo blkid
/dev/sda2: LABEL="ROOT" UUID="de0eff4f-9c54-48cc-8a32-644ae55d2f47" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="ROOT" PARTUUID="e3679d41-ba89-d74a-9160-c3f0bd84991a"
/dev/sda3: UUID="324269e3-e5d1-4f31-afbb-97201a0ec2e5" TYPE="swap" PARTLABEL="SWAP" PARTUUID="0d1c3272-8077-f147-aab1-372720438cfb"
/dev/sda1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="E0EA-BFF6" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="BOOT" PARTUUID="8a3f2bc0-3675-ab4e-8117-54ecae5064a2"

Once more: Thanx & regards
Werner

seth · 2022-12-08 07:42:26

Depends on you bootloader, https://wiki.archlinux.org/title/Kernel_parameters

tronics · 2022-12-08 11:12:25

seth wrote:

Depends on you bootloader, https://wiki.archlinux.org/title/Kernel_parameters

I changed the loader file, but there is still a 90 second hang... this time searching both partitions via uuid.

Here are my loader files:

[werner@werners-E756 ~]$ cat /boot/loader/loader.conf 
#timeout 3
#console-mode keep
#


default  arch-uefi.conf
timeout  2
console-mode max
editor   no
[werner@werners-E756 ~]$ cat /boot/loader/entries/arch-uefi.conf 
title    Arch Linux
linux    /vmlinuz-linux
initrd   /intel-ucode.img
initrd   /initramfs-linux.img
# options  root=LABEL=ROOT rw lang=en init=/usr/lib/systemd/systemd locale=en_US.UTF-8
options  root=UUID=de0eff4f-9c54-48cc-8a32-644ae55d2f47 rw lang=en init=/usr/lib/systemd/systemd locale=en_US.UTF-8

[werner@werners-E756 ~]$

Again: Thanx & regards
Werner

seth · 2022-12-08 13:02:50

Dez 02 10:30:59 werners-E756 systemd[1]: Finished File System Check on /dev/disk/by-label/ROOT.
…
Dez 02 10:32:39 werners-E756 systemd[1]: Timed out waiting for device /dev/disk/by-label/ROOT.

So the FS is checked and then 2:10 minutes later we time out waiting for it…
Since you're using systemd-boot and there's been a flurry of "systemd-boot delay" posts, can you try to downgrade systemd to 251.7-4? (earlier versions will get you into trouble b/c of the openssl upgrade)
https://wiki.archlinux.org/title/Arch_Linux_Archive

tronics · 2022-12-14 10:53:05

seth wrote:

Dez 02 10:30:59 werners-E756 systemd[1]: Finished File System Check on /dev/disk/by-label/ROOT.
…
Dez 02 10:32:39 werners-E756 systemd[1]: Timed out waiting for device /dev/disk/by-label/ROOT.
So the FS is checked and then 2:10 minutes later we time out waiting for it…
Since you're using systemd-boot and there's been a flurry of "systemd-boot delay" posts, can you try to downgrade systemd to 251.7-4? (earlier versions will get you into trouble b/c of the openssl upgrade)
https://wiki.archlinux.org/title/Arch_Linux_Archive

I just downgraded to systemd 251.7-4!!!

And it really works!

Question: Any idea, when I may update again?

Thank you very much & regards
Werner

seth · 2022-12-14 13:06:32

Apparently it's gotten worse - at leat in some cases: https://github.com/systemd/systemd/issues/25737
There're also other bootloaders …

Arch Linux

#1 2022-12-02 10:30:51

after update system hangs for 90sec and needs extra ^d key ...

#2 2022-12-02 10:40:59

Re: after update system hangs for 90sec and needs extra ^d key ...

#3 2022-12-02 11:43:15

Re: after update system hangs for 90sec and needs extra ^d key ...

#4 2022-12-02 14:25:39

Re: after update system hangs for 90sec and needs extra ^d key ...

#5 2022-12-02 16:53:20

Re: after update system hangs for 90sec and needs extra ^d key ...

#6 2022-12-03 05:10:12

Re: after update system hangs for 90sec and needs extra ^d key ...

#7 2022-12-06 11:26:15

Re: after update system hangs for 90sec and needs extra ^d key ...

#8 2022-12-06 11:34:11

Re: after update system hangs for 90sec and needs extra ^d key ...

#9 2022-12-06 11:46:24

Re: after update system hangs for 90sec and needs extra ^d key ...

#10 2022-12-06 15:02:40

Re: after update system hangs for 90sec and needs extra ^d key ...

#11 2022-12-08 07:09:48

Re: after update system hangs for 90sec and needs extra ^d key ...

#12 2022-12-08 07:42:26

Re: after update system hangs for 90sec and needs extra ^d key ...

#13 2022-12-08 11:12:25

Re: after update system hangs for 90sec and needs extra ^d key ...

#14 2022-12-08 13:02:50

Re: after update system hangs for 90sec and needs extra ^d key ...

#15 2022-12-14 10:53:05

Re: after update system hangs for 90sec and needs extra ^d key ...

#16 2022-12-14 13:06:32

Re: after update system hangs for 90sec and needs extra ^d key ...

Board footer