SOLUTION I should have never removed UUIDs from my fstab, instead I should have ADDED them to my bootloader config. Putting UUIDs back in fstab and adding the UUID for my root partition to syslinux.cnf solved this.
I've been pruning my initramfs following falconindy's blog post which I learned about from success stories in the systemd startup times thread.
I ended up with this mkinitcpio.conf
MODULES="ahci sd_mod ext4" BINARIES="fsck fsck.ext4" HOOKS="base consolefont" COMPRESSION="lzop"
I've also tried without the compression method specified. I've added in a couple modules which I admittedly don't know much about, but I found them in the lsmod list after appending the "break=postmount" to a working initramfs. These included uhci_hcd and ehci_hcd.
In every case I get the same error message on boot:
device '/dev/sda3' not found. Skipping fsck Unable to find root device '/dev/sda3'
and I am dropped to a recovery shell. There is a message that says (paraphrasing) "type exit to attempt to continue". I type exit and hit enter and I get a message:
Trying to continue (This will likely fail)
But it does not fail - it works perfectly and the system boots successfully.
So in short - everything works, but there is some early boot test that "thinks" it won't work and leads to me having to exit from a recovery shell.
I realized my fstab still had UUIDs, so I changed those to /dev/sdaX and rebuilt the initramfs, but this had no effect.
The best I can guess (and it is a wild guess) is that the only remaining difference between the working initramfs and this problematic stripped down version is the absence of udev. I did look at a diff of the lsinitcpio of a working version and the failing trimmed down version. I've removed any items that went both ways in the diff output (eg ">some/file ... <some/file") and this is what remains:
7c6 < ./hooks/udev --- 16,18d14 < ./usr/bin/fsck.ext3 20,21c16,17 < ./usr/bin/udevadm < ./usr/bin/udevd --- 130a127,128 > ./usr/lib/libpthread-2.16.so > ./usr/lib/libpthread.so.0 137,151d134 < ./usr/lib/udev < ./usr/lib/udev/scsi_id < ./usr/lib/udev/ata_id < ./usr/lib/udev/rules.d < ./usr/lib/udev/rules.d/80-drivers.rules < ./usr/lib/udev/rules.d/60-persistent-storage.rules < ./usr/lib/udev/rules.d/50-udev-default.rules < ./usr/lib/libpthread-2.16.so < ./usr/lib/libpthread.so.0 < ./usr/lib/libattr.so.1.1.0 < ./usr/lib/libattr.so.1 < ./usr/lib/librt-2.16.so < ./usr/lib/librt.so.1 < ./usr/lib/libacl.so.1.1.0 < ./usr/lib/libacl.so.1 254a238,246 > ./usr/lib/modules/3.5.3-2-ck/kernel/hwa-hc.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/isp116x-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/isp1362-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/isp1760.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/ohci-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/oxu210hp-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/r8a66597-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/sl811-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/u132-hcd.ko 255a248,250 > ./usr/lib/modules/3.5.3-2-ck/kernel/whci-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/xhci-hcd.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/ftdi-elan.ko 256a252,256 > ./usr/lib/modules/3.5.3-2-ck/kernel/wusb-wa.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/wusbcore.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/umc.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/uwb.ko > ./usr/lib/modules/3.5.3-2-ck/kernel/whci.ko 274,275d273 < ./etc/udev < ./etc/udev/udev.conf
The ck-modules lead me to try this with a "vanilla" arch kernel which I also have installed. With the default arch kernel I am left in a recovery shell but exiting does not lead to a successful boot. I don't know what this really means, but perhaps something unique to the ck-kernel setup allows a recovery from this problem.
Any suggestions for further troubleshooting or exploration?
edit I put udev back in the hooks array, rebuilt the image, and now it works fine. So udev is the relevant peice of the puzzle. I thought the idea of using the modules, though, was so that udev wouldn't be needed. I must have missed a needed module, though I cant figure what it would be.
edit2 I've used "break=postmount" on both the working image with udev, and with the failing image without udev and I do an "lsmod" in the shell. I get the exact same list of modules. So all the correct modules are all loaded properly. But the absence of udev seems to cause a problem.
Last edited by Trilby (2012-09-01 15:10:14)