You are not logged in.

#1 2013-05-30 03:43:49

frankspace
Member
Registered: 2009-09-19
Posts: 12

[SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

After a couple of years or so of working perfectly, today one of my arch computers simply stopped booting all the way.  After selecting which kernel to use, I get this:

:: running early hook [udev]
udevd: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
:: running early hook [lvm2]
:: running hook [udev]
udevadm: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
udevadm: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
udevadm: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
:: running hook [encrypt]
Waiting 10 seconds for device /dev/sda2 ...
Waiting 10 seconds for device /dev/mapper/gcg-root ...
ERROR: device '/dev/mapper/gcg-root' not found. Skipping fsck.
ERROR: Unable to find root device '/dev/mapper/gcg-root'.
You are being dropped to a recovery shell
  Type 'exit' to try and continue booting
sh: can't access tty; job control turned off
[rootfs /]#

I tried booting from grml and chrooting using more or less what's described here:  http://mzanfardino.wordpress.com/2012/0 … rch-linux/ with a few modifications for the specific lvm setup I've been using.  Pleasingly, I found this surprisingly easy!  I used pacman to reinstall zlib, udev, mkinitcipo, linux, and linux-headers; ran mkinitcipo; and rebooted.  Unfortunately, then the same thing happened.  So I tried again, and this time, because I was still using grub-legacy, I upgraded to grub2.  That seemed to work, but I rebooted again, and once again got the same issue:  libz.so.1 cannot be found, followed by my encrypted LVM partition failing to be found.

So I'm stumped.  Googling for missing libz does not seem to give me any results which which I can interpret as helpful.  The recovery shell doesn't seem to do much.  In fact, some of the commands (I tried modprobe out of curiosity) also run into the lack of libz.  I have no idea.  From inside the chroot, there really does seem to be a perfectly good copy of /lib/libz.so.1.2.8, properly symlinked to /lib/libz.so.1.  In fact, everything inside the chroot seemed to work flawlessly.  I don't, however, know if there is any other diagnostic test for which I should hunt.

All of my data is backed up, so nuking and paving is technically an option.  But it's a really, really annoying one, and it'd be a collossal waste of time that I don't really want to spend.  This has got to be recoverable without that, surely?  What would the gurus suggest I do next?  Thank you!

Last edited by frankspace (2013-05-31 03:30:19)

Offline

#2 2013-05-30 08:49:25

mich41
Member
Registered: 2012-06-22
Posts: 796

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

It seems that mkinitcpio screwed up and didn't include libz (or some dependency) in the image. Can you find libz.so anywhere from this "recovery shell"?

I'd start with wiki article on mkinitcpio, check mkinitcpio.conf, etc.

Offline

#3 2013-05-30 11:34:48

frankspace
Member
Registered: 2009-09-19
Posts: 12

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

Nope, there does not appear to be a libz.so anywhere within the recovery shell.  I'll check into mkinitcpio when I get home from work, thanks!  But I did actually double-check my mkinitcpio.conf when I ran mkinitcpio -p linux, and it appeared fine to me as far as I could tell.  I'd been lazy and hadn't previously removed an obsolete hook that I am suddenly unable to remember but that hadn't been causing me problems previously, so I removed it, which is why I was in mkinitcpio.conf; it didn't seem to make a difference, though.  I'm puzzled by why it would suddenly break when it's been working so far, but a b0rked initial ramdisk does seem like the sanest starting point.  Thank you!

Incidentally, I forgot to mention previously:  selecting the fallback image results in exactly the same behaviour.

Offline

#4 2013-05-30 12:30:10

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

I'm more interested in what you did before this broke. You can't possibly claim that you did nothing and this suddenly changed.

Offline

#5 2013-05-30 13:12:33

frankspace
Member
Registered: 2009-09-19
Posts: 12

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

Indeed not, and you're right, I forgot to mention that.  Sorry!  I did a "sudo pacman -Syu" and then rebooted afterwards.  Linux was one of the upgraded packages.  I did not, however, see pacman or mkinitcpio generate any errors or warnings aside from the above-mentioned depreciated hook that I'd been lazily not removing from mkinitcpio.conf.  Unfortunately, I do not recall which other packages were updated other than util-linux.  (I upgraded no aur packages at that time.)  In the chroot I tried downgrading util-linux to the previous version, but doing so had no effect on this problem.  It's possible I failed to notice something, of course.  But my other computer (the one I'm using right now) is using the same kernel version as the one to which my broken computer was upgraded.  I haven't touched pacman on the working computer since the breakage, though.  Are there any specific logs I should look into?  I'm probably not as clueful about logs as I really ought to be.  Thanks!

Offline

#6 2013-05-30 13:17:11

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,041

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

Try /var/log/pacman.log for starters.

Offline

#7 2013-05-30 13:49:06

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

I suspect the output of 'ls -l /' (where / is your real root) is relevant as well as 'lsinitcpio -a' for the initramfs.

Offline

#8 2013-05-31 03:38:15

frankspace
Member
Registered: 2009-09-19
Posts: 12

Re: [SOLVED] Can't boot udev can't find libz.so or encrypted lvm partition

I solved it!  There were actually two problems, neither of which I'd found without the hint of lsinitcpio.  (The pacman log and the contents of my root directory were uninteresting, but now I know how to check my pacman log!)

I did lsinitcpio -a, -l, and -c on both the broken computer and on my working computer, and compared them.  -c showed that I was using identical configuration files.  -a showed a few irrelevant module differences that seemed to be due to one computer being a desktop and the other a laptop.  But -l showed that libz.so was indeed present, but in the directory /opt/brlcad/lib.  And indeed, I'd recently installed brlcad from AUR, and as it turns out, brlcad stuffs a bunch of stuff into /opt, including an old version of zlib.  For some reason, mkinitcpio was picking this up.  I hadn't gotten around to trying brlcad out yet, so I just uninstalled it and re-ran mkinitcpio -p linux.  No more problem with not being able to find libz.so!

However, I still couldn't boot.  It turned out that in addition, grub had removed any reference to my cryptdevice.  I don't know why, because I didn't upgrade from grub-legacy to grub2 until I was already getting this problem.  Nevertheless, between the wiki and being able to look at what I'd used in my backup grub configuration, it was easy enough to put the correct cryptdevice reference back into /etc/grub/default and regenerate grub.  And now I can boot fine, and am, in fact, typing this on the previously broken computer.

There was a lot of trial-and-error here, as well, but I left that out of this story.  Thanks for your help, everyone!

Offline

Board footer

Powered by FluxBB