Arch Linux

grepfor

Member

Registered: 2012-09-19

Posts: 28

Delayed appearance of /dev/disk/by-label

Been trying to figure out why fsck during initrd does not find the root device, when root is specified by LABEL=xxxx in the kernel parameters. The source of the problem appears to be that, for reasons not understood by me, the "by-label" subdirectory of /dev/disk does not exist at the time the fsck script is looking for it. But have not been able to make any further progress in figuring out why that is. 

Attempts to investigate using the premount shell have led nowhere because it seems that dropping to the premount shell masks the problem and it doesn't occur.

Here's the observed behavior without break=premount:

    :: running early hook (udev)
    starting version 239
    :: running hook (udev)
    :: Triggering uevents...
    Waiting 10 seconds for device /dev/disk/by-label/rootski ...
    ERROR: device 'LABEL=rootski' not found. Skipping fsck.
    :: mounting 'LABEL=rootski' on real root
    :: running cleanup hook (udev)

    (boot continues normally)

Here's the behavior when specifying "break=premount" and then just typing "exit" as soon as the premount shell prompt appears:

    :: running early hook (udev)
    starting version 239
    :: running hook (udev)
    :: Triggering uevents...
    :: Pre-mount break requested, type 'exit' to resume operation
    sh: can't access tty: job control turned off
    [rootfs ] # exit
    :: performing fsck on '/dev/sda5'
    rootski: clean, xxxx/xxxxx files, xxxx/xxxx blocks
    :: mounting '/dev/sda5' on real root
    :: running cleanup hook (udev)

    (boot continues normally)

In subsequent attempts to diagnose the situation from the premount shell, I noticed that doing

ls -l /dev/disk

immediately after the premount shell prompt lists only the "by-id" and "by-path" subdirectories, i.e. the "by-label", "by-uuid", and "by-partuuid" subdirs are missing. However, if I keep doing "ls -l /dev/disk", those subdirs do eventually appear.  The period of non-existence varies from boot to boot, but usually somewhere from 20 seconds to a minute or so.

So it seems that the fsck script is telling the truth, in the sense that the root device, when searched for by label, really doesn't exist at the time it is looking for it. But how to go about figuring out why the appearance of /dev/disk/by-label is delayed like this?

Last edited by grepfor (2019-01-13 21:30:47)

dmartins

Member

Registered: 2006-09-23

Posts: 360

Re: Delayed appearance of /dev/disk/by-label

Does this still happen with the latest systemd packages?
The by-label symlinks are created by udev. You can enable debug logging in udev by setting udev_log=debug in /etc/udev/udev.conf. Then rebuild your initramfs with mkinitcpio -P.

If it still occurs with the latest systemd, please include the debug logs, as well as the output from blkid.

grepfor

Member

Registered: 2012-09-19

Posts: 28

Re: Delayed appearance of /dev/disk/by-label

Thanks for your response, Dan.

Re: systemd version: Systemd version is 240.32.2, which appears to be  'latest' as of last Syyuv a few days ago.

Re: udev logging: I had previously tried dumping udev events via kernel parm udev.log-priority=debug, but tried what you suggested above anyway, thinking perhaps it might provide different or additional info. But it seems to be dumping the same stuff as with the kernel parm approach. (Btw, as an aside: In order for changes in /etc/udev/udev.conf to be effective during initrd, it's evidently also necessary to add /etc/udev/udev.conf to the FILES() hook in /etc/mkinitcpio.conf. Otherwise it doesn't show up in the initram image.)

But here's the thing about the udev debug output: So much debug info is driven to the console that the realtime consumed by that spewage winds up heisenberging  the problem:  /dev/disk/by-label (and friends) show up at the time the probing is done in debug mode, so the root fsck winds up being done as expected. But, if I remove the debug dumping, the probing is evidently done much earlier, and the "by-label" link hasn't shown up by that time, so the root fsck gets skipped.

Do you know of any way to force udev debug output only to a file during initrd  (say, on /run) where it can be viewed in the premount shell? Avoiding the time-consuming console spewage might be the only way to see what udev is actually doing that leads to the non-creation of the by-label (etc) links.

Any other suggestions of any type welcomed. Thanks again for your time.

- G

dmartins

Member

Registered: 2006-09-23

Posts: 360

Re: Delayed appearance of /dev/disk/by-label

Ahh, yes. That seems to be a common theme with udev troubleshooting!

I didn't know about the kernel param for setting udev's log level - thanks for that. It took some screwing around, but I think I found a working set of kernel parameters that result in udev's debug messages being captured in the journal & dmesg. I hope this works for you. You can get the logs from dmesg or journalctl (-k) after boot.

rd.udev.log_priority=debug
rd.log=kmsg
printk.devkmsg=on
log_buf_len=1M

rd.log is from mkinitcpio and tells the initramfs to use kmsg for logging. There is also a 'file' option that should send the logs to /run/initramfs/init.log, but it doesn't capture udev's messages for some reason.
printk.devkmsg=on - this disables rate limiting for printk. Without it, most udevd messages get dropped
log_buf_len sets the kernel ring buffer. Arch's default appears to be 128k, which results in the earlier messages being overwritten.

grepfor

Member

Registered: 2012-09-19

Posts: 28

Re: Delayed appearance of /dev/disk/by-label

Thanks for the legwork on this, Dan. Your combo of kernel parms did indeed lead to a  boot in which nearly all console yapping is suppressed, and the udev actions that underlie the problem wind up in the journal, unimpaired by the console delays punching holes in the realtime.

That should give me enough info to get to the next stage myself of figuring out why the "by-XXXX" symlinks aren't showing up. Whether I can solve that I'm not sure yet, but at least the necessary raw material for debug is now available, thanks to your suggestions.

Tip o' the hat for your help on it, much appreciated!