[SOLVED] /dev/disk/by-path and others missing during initial boot

thetrivialstuff · 2010-05-17 01:25:19

I'm not sure what's causing this, but here's what I've got:

After upgrading to 2.6.33 (and upgrading udev to 151-3), the mkinitcpio update ran as usual, no errors, and the RAID hook reported using my custom config file, as it should. Then when I rebooted, it reached the mdadm hook and stopped, waited 10 seconds for /dev/md2 (my root filesystem), then bailed into the recovery shell. It was as if mdadm didn't do anything at all; no arrays were assembled.

The immediate cause is that all of my arrays are specified by using persistent device names from /dev/disk/by-path (for a long rant about why it's that and not UUID or some other method, see: http://bbs.archlinux.org/viewtopic.php?id=81674), and /dev/disk/by-path isn't being created. The udev rules to create it exist and are on the ramdisk image, but udev apparently isn't processing them. But, I have no idea what would cause it not to do that (no errors before the mdadm hook) or how to fix it.

Any ideas?

~Felix.

Last edited by thetrivialstuff (2010-05-17 06:55:34)

fphillips · 2010-05-17 05:34:24

You have no symlinks under /dev/disk/by-path, but the ones for by-id and by-uuid are there?

udevadm test /sys/block/sda &> outfile
grep -i path outfile |less -S

Here's mine for comparison: http://pastebin.com/GQ0XAzn9

Then make sure sysfs is getting populated, for instance:
/sys/devices/pci0000\:00/0000\:00\:1f.2/host0/target0\:0\:0/0\:0\:0\:0/block/sda/

Last edited by fphillips (2010-05-17 05:45:50)

thetrivialstuff · 2010-05-17 05:46:36

the whole /dev/disk/by-path directory doesn't exist -- the only subdir of /dev/disk is by-uuid.

But that's only in the initial ramdisk -- after the root filesystem comes up (which I've been able to do by editing the mdadm.conf file in the ramdisk's recovery shell, so that it uses /dev/sda and /dev/sdb instead, and then re-running mdassemble), I have all of /dev/disk/{by-id,by-path,by-uuid}. That (I think) lets the rest of the RAID arrays come up during "Activating RAID arrays" -- I previously had them assemble during the initial ramdisk stage as well.

As the problem is during initial ramdisk, I don't know a good way of getting a dump of udevadm test -- but I was playing around with it and the weird thing was that it *was* processing the "/lib/udev/rules.d/60-persistent-storage.rules" file (but apparently not triggering the rules that made the by-path entries). When I run that command on my fully booted system, I see about what you posted, including the by-path entries.

I don't suppose you can cause a deliberate crash during initial ramdisk and see if your copy of udev in that part of the boot process creates /dev/disk/by-path? (Am thinking of trying that on my laptop -- will post the results.)

~Felix.

thetrivialstuff · 2010-05-17 06:23:09

OK, just tried it on the laptop, and the same thing happened -- it's looking more like a bug somewhere in mkinitcpio and/or the udev package.

I forced a crash into init ramdisk recovery shell (by setting my rootfs to /dev/narf in grub ), and I don't have /dev/disk/by-path on here either. (I checked before I did the upgrade and /dev/disk/by-path existed in the previous ramdisk I had on here, which was from kernel 2.6.31.)

I guess this could still be caused by me not upgrading everything at once (on the desktop I wanted to update the kernel first and then the other stuff, because I wasn't sure I'd have enough disk space for pacman -Syu, and on the laptop I updated just the kernel / mkinitcpio because I was testing this) -- I'll see what happens when I bring the laptop completely up-to-date.

Could you try the force crash trick and see if you have /dev/disk/by-path in your initial ramdisk? Just hit 'e' in grub and change your root filesystem to something nonexistent. That'll force the crash, then do ls /dev/disk and see if by-path is there, and reboot with ctrl+alt+del to boot normally again.

~Felix.

fphillips · 2010-05-17 06:24:13

I'm not seeing any of the *_id binaries from the udev package in initrd, and the udev rule needs path_id. Try sticking that in there and see if it works. I can give you the steps for that if you need it.

fphillips · 2010-05-17 06:28:48

Nope no by-path!

Late night hacking is the best...

fphillips · 2010-05-17 06:34:59

Ah cool, BINARIES= in mkinitcpio.conf. In Slackware I would have to gunzip and loop-mount the initrd to add a file.

thetrivialstuff · 2010-05-17 06:53:39

Yay, that fixed it!

...I can't believe I was the only one using by-path in my bootup

Gonna go file a bug report now; thanks for the help with this, it would've taken me at least most of tomorrow to figure this out (didn't know / read things carefully enough to know that udev needed extra binaries to make those symlinks happen).

~Felix.

fphillips · 2010-05-17 06:59:58

Cool, thanks - I don't get to work on something like this very often.

It looks like no usb/ata/scsi persistent rules will work in the initrd w/o *_id. That's just dumb, because that's when they're typically needed.
I guess this is a bug for the mkinitcpio package, though Thomas may just tell you to use BINARIES=.

Frank

thetrivialstuff · 2010-05-17 07:10:00

There seems to be a conspiracy against my particular system and its use of software RAID, I tell ya.

Way back when hard drives were /dev/hda, /dev/hdb, etc., the names *were* persistent (and a lot easier to type)... then SATA came along, and it still worked great for a few months -- the IDE drives were named like that, and first SATA was /dev/sda, second SATA /dev/sdb, etc.

Then they changed to /dev/sd* naming across the board and it all went bonkers -- my IDE drives would get into races with my SATA drives to see who could get /dev/sda first, and gods forbid I left an external drive plugged in (for they get sd* names too!).

Use UUID and auto-assembly, they said... only UUID relies on the drive to identify itself, which has always struck me as a very bad idea (see my rant thread :P).

So, I finally switched to persistent device naming a la by-path, because I was getting tired of having to re-apply hacks to keep /dev/hd* naming every time I upgraded the kernel... but alas, it seems that even now my RAID arrays are still doomed to break on pacman -Syu. Oh well :)

Edit: I meant to say in the rambling above that I've often had to do exactly this kind of poking around in the initrd image -- but because persistent device naming worked for the last couple upgrades, I'm out of practice ;) (I think I'll go to sleep now; it's late over here...)

~Felix.

Last edited by thetrivialstuff (2010-05-17 07:22:12)

Arch Linux

#1 2010-05-17 01:25:19

[SOLVED] /dev/disk/by-path and others missing during initial boot

#2 2010-05-17 05:34:24

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#3 2010-05-17 05:46:36

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#4 2010-05-17 06:23:09

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#5 2010-05-17 06:24:13

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#6 2010-05-17 06:28:48

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#7 2010-05-17 06:34:59

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#8 2010-05-17 06:53:39

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#9 2010-05-17 06:59:58

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

#10 2010-05-17 07:10:00

Re: [SOLVED] /dev/disk/by-path and others missing during initial boot

Board footer