Arch Linux

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Initrd boot fails with systemd 242.0-1

lvm2 2.02.184-4
device-mapper 2.02.184-4

Reverting to systemd 241.93-1 and rebuilding the initrd works.  Booting the same kernel but with an old initrd also works.

The hang appears to be before the root filesystem is mounted, so I'm not sure how I can get a bootlog.

loqs

Member

Registered: 2014-03-06

Posts: 18,123

Re: Initrd boot fails with systemd 242.0-1

What is the contents of /etc/mkinitcpio.conf?  If you remove quiet from the boot parameters what is the last output before it fails?

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

If that looks minimal - it is.  I build a kernel with just the modules I care about build into the kernel.

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base systemd autodetect fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
COMPRESSION_OPTIONS=("-9")

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

https://photos.app.goo.gl/9xM74H7QZknHv5SW8

loqs

Member

Registered: 2014-03-06

Posts: 18,123

Re: Initrd boot fails with systemd 242.0-1

From the top of the image boot reaches initrd.target then lots of service shutdowns as expected but it does not reach initrd-switch-root.target

If you add the debug option as well as removing quiet a lot more output should be produced.

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

Here's a boot on another computer, similarly configured, with quiet off and with debug on:

https://photos.app.goo.gl/wurMtKLNaAVDDDRx7

(Same result - hangs here with systemd 242, boots fine with older systemd)

device-mapper 2.02.184-4
lvm2 2.02.184-4
systemd 242.0-1

Last edited by DebauchedSloth (2019-04-12 22:46:17)

loqs

Member

Registered: 2014-03-06

Posts: 18,123

Re: Initrd boot fails with systemd 242.0-1

The link in post #6 does not work for me.

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

Sorry.  My bad.

https://photos.app.goo.gl/gEW4vKWV9pFtRKNd8

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

Here's the mkinitcpio.conf from computer #2, BTW:

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run. Advanced users may wish to specify all system modules
# in this array. For instance:
#   MODULES="piix ide_disk reiserfs"
MODULES=""

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image. This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=""

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way. This is useful for config files.
#FILES="/usr/lib/firmware/amdgpu/polaris12_ce.bin /usr/lib/firmware/amdgpu/polaris12_mc.bin /usr/lib/firmware/amdgpu/polaris12_me.bin /usr/lib/firmware/amdgpu/polaris12_mec2.bin /usr/lib/firmware/amdgpu/polaris12_mec.bin /usr/lib/firmware/amdgpu/polaris12_pfp.bin /usr/lib/firmware/amdgpu/polaris12_rlc.bin /usr/lib/firmware/amdgpu/polaris12_sdma1.bin /usr/lib/firmware/amdgpu/polaris12_sdma.bin /usr/lib/firmware/amdgpu/polaris12_smc.bin /usr/lib/firmware/amdgpu/polaris12_uvd.bin /usr/lib/firmware/amdgpu/polaris12_vce.bin"

# HOOKS
# This is the most important setting in this file. The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added. Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##  This setup specifies all modules in the MODULES setting above.
##  No raid, lvm2, or encrypted root is needed.
#  HOOKS="base"
#
##  This setup will autodetect all modules for your system and should
##  work as a sane default
#  HOOKS="base udev autodetect block filesystems"
#
##  This setup will generate a 'full' image which supports most systems.
##  No autodetection is done.
#  HOOKS="base udev block filesystems"
#
##  This setup assembles a pata mdadm array with an encrypted root FS.
##  Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#  HOOKS="base udev block mdadm encrypt filesystems"
#
##  This setup loads an lvm2 volume group on a usb device.
#  HOOKS="base udev block lvm2 filesystems"
#
##  NOTE: If you have /usr on a separate partition, you MUST include the
#  usr, fsck and shutdown hooks.
HOOKS="base systemd autodetect modconf block filesystems consolefont"

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS="-T 8"
COMPRESSION_OPTIONS="-9"

loqs

Member

Registered: 2014-03-06

Posts: 18,123

Re: Initrd boot fails with systemd 242.0-1

Could you post a close up of the bottom part of the screen?  There looks to be a message systemd-journald (possibly after a timeout) but I can not make it out.
Does either system have a serial port?
If not you could bisect between 242 and 241.93 or report the issue upstream with what you have.

raneon

Member

Registered: 2013-11-02

Posts: 57

Re: Initrd boot fails with systemd 242.0-1

Just wanted to add, I can't boot with systemd 242 either.

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

Sorry thought I had done this earlier.  The last line is systemd-journald sent WATCHDOG-1 notification - and it did come in later.

I can see about getting a serial port in one of them - what would I do with that.

loqs

Member

Registered: 2014-03-06

Posts: 18,123

Re: Initrd boot fails with systemd 242.0-1

You would need two systems with serial ports or a virtual machine then you might be able to get more output following [1]
Alternatively I can provide instructions to bisect systemd which should hopefully locate the first bad commit.
You could also look through [2] see if you can find a matching issue I tried looking but may have missed it.

[1] https://freedesktop.org/wiki/Software/s … Debugging/
[2] https://github.com/systemd/systemd/issues

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

OK, thanks.  I'm not sure there is anymore debug logging to pull over the serial interface, but maybe.  I'll see if I can locate a serial card around here.

I had looked at the systemd issues and didn't see anything close. 

If you can give me instructions for bisecting systemd, that would be great.  I've done this many times with the kernel in the past, but never with systemd.

loqs

Member

Registered: 2014-03-06

Posts: 18,123

Re: Initrd boot fails with systemd 242.0-1

The following assumes the base-devel group and git are installed.

$ git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/systemd"
$ cp -r packages/trunk systemd
$ rm -rf packages
$ makepkg -rsi #this is to confirm systemd as built on your system has the issue

$ cd systemd/src/systemd-stable
$ git bisect start
$ git bisect bad
$ git reset --hard
$ git checkout 3d835d09f29151c97af7cb1116e192df711587ab
$ git apply ../../0001-Use-Arch-Linux-device-access-groups.patch
$ rm -rf ../build
$ cd ../..
$ makepkg -ersi #this is to confirm systemd 241.91 as built on your system does not have the issue

$ cd systemd/src/systemd-stable
$ git reset --hard
$ git bisect good
$ rm -rf ../build
$ cd ../..
$ makepkg -ersi #test the bisection point

$ cd systemd/src/systemd
$ git reset --hard
$ git bisect $result #substitute good or bad here
$ git apply ../../0001-Use-Arch-Linux-device-access-groups.patch
$ rm -rf ../build
$ cd ../..
$ makepkg -ersi #repeat these seven lines and test the generated package until git has found the bad commit

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

OK, will try.  That's going to take a while.

So, I have now tried this on four systems.  Two desktops, two notebooks.  Both notebooks work with 242, both desktops fail.  One interesting thing is that the notebooks are single drive systems and the desktops both have two.  I'm wondering if that may be relevant.

fscan

Member

Registered: 2019-04-14

Posts: 1

Re: Initrd boot fails with systemd 242.0-1

FYI, i had the same issue.
Replacing the 'systemd' mkinitcpio hook with the 'udev' one and rebuilding the initrd fixed it for me.
It seems like the hook broke with the systemd upgrade.

DebauchedSloth

Member

Registered: 2014-12-08

Posts: 29

Re: Initrd boot fails with systemd 242.0-1

FYI, i had the same issue.
Replacing the 'systemd' mkinitcpio hook with the 'udev' one and rebuilding the initrd fixed it for me.
It seems like the hook broke with the systemd upgrade.

Same here - replacing the systemd hook with udev does indeed fix the problem!  Thanks!

I also tried removing my secondary drive - the problem persisted, although I did see one successful boot with debug on.  Some sort of race condition.

brittyazel

Member

From: Davis, CA

Registered: 2013-05-11

Posts: 163

Re: Initrd boot fails with systemd 242.0-1

Just an FYI, it would have been awesome of someone filed a bug report when this was figured out. Unfortunately now we have the broken package in stable, and the packagers were caught off guard

---

I don't really know what I'm doing.

brli7848

Member

Registered: 2015-04-26

Posts: 3

Re: Initrd boot fails with systemd 242.0-1

Same issue, the hook rename trick doesn't help, downgrade to 241.93 and IgnorePkg it for now.

Last edited by brli7848 (2019-04-24 12:08:58)

blispx

Member

Registered: 2017-11-29

Posts: 53

Re: Initrd boot fails with systemd 242.0-1

I don't think they'll fix it ;-(

I've had the same for a year

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,430

Re: Initrd boot fails with systemd 242.0-1

Please don't necrobump old threads with a vanity post, if you want to actually look at this, open up a new thread with details on what fails where and how.

https://wiki.archlinux.org/index.php/Co … mpty_posts
https://wiki.archlinux.org/index.php/Co … bumping%22

Closing.