Arch Linux

DcUK

Member

Registered: 2009-04-09

Posts: 10

[SOLVED] LVM2/device-mapper upgrade 2.02.103-1 renders PC unbootable

Hi,

After an full system update last night of a number of packages (including kernel 3.11.5-1, dbus 1.6.16-1, lvm2 and device-mapper 2.02.103-1 but not systemd), my PC no longer boots. It loads the grub2 menu, and begins to boot normally, but hangs with systemd waiting for the disks to become ready. The partitions in question are LVM2 volumes on top of a software RAID array. The root partition (containing /usr and /var) is a seperate physical disk partition and is correctly detected. My fstab looks like this:

# 
# /etc/fstab: static file system information
#
# <file system>        <dir>                    <type>  <options>                                  <dump> <pass>
LABEL=ROOT             /                        ext3    defaults                                   0      1
LABEL=HOME             /home                    ext4    defaults                                   0      2
LABEL=ANNA             /home/anna               ext4    defaults                                   0      2
LABEL=BACKUP           /home/backup             ext4    defaults                                   0      2
LABEL=SRV              /srv                     ext4    defaults,nosuid                            0      2

and my HOOKS looked like this (and the initrd has been rebuilt after the updates):

HOOKS="base systemd autodetect modconf block mdadm_udev filesystems keyboard fsck keymap"

My first thought was a problem with the systemd HOOK, although this has worked correctly until now. I booted a rescue disk, chroot, and rebuilt the initrd with the following HOOKS:

HOOKS="base udev autodetect modconf block mdadm_udev filesystems keyboard fsck keymap"

This still exhibited the exact same problem, all mounts with the exception of ROOT fail to work. To compound the issue, the boot process fails and tries to drop me into a shell (offering the password or Control-D options), but whatever I type at this point fails the password check making further diagnosis a challenge.

I looked through the other packages that had been updated and tried rolling back both lvm2 and device-mapper to 2.02.100-1, rebuilt the initrd, and rebooted. This worked perfectly. To rule out other changes, I also rolled back the HOOKS array to the first one listed above (including the systemd hook) and rebuilt once again. This also booted successfully.

To summarize: With lvm2 and device-mapper 2.02.103-1 my PC fails to boot with or without the systemd HOOK, but with 2.02.100-1 both options work successfully.

The only uncommented lines in my mdadm.conf are:

DEVICE partitions
ARRAY /dev/md1 metadata=1.2 name=epitome:raid1-lvm-data UUID=16932334:598d3612:6a57b532:88cbf000

The RAID array is RAID1 consisting of a single partition on each of two disks:

dev/md1:
        Version : 1.2
  Creation Time : Tue Aug 24 20:27:02 2010
     Raid Level : raid1
     Array Size : 976759672 (931.51 GiB 1000.20 GB)
  Used Dev Size : 976759672 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Fri Oct 18 12:19:17 2013
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : epitome:raid1-lvm-data  (local to host epitome)
           UUID : 16932334:598d3612:6a57b532:88cbf000
         Events : 8955

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       2       8       33        1      active sync   /dev/sdc1

Is anybody else seeing problems with this release? Or know of anything that changed that might have caused this behaviour?

Last edited by DcUK (2013-10-18 14:04:59)

DcUK

Member

Registered: 2009-04-09

Posts: 10

Re: [SOLVED] LVM2/device-mapper upgrade 2.02.103-1 renders PC unbootable

Well, upon further investigation it relates to the LVM partitions. With lvm2 and device-mapper 2.02.100-1 the system boots reliably without the lvm2 HOOK, but from 2.02.103-1 it requires the lvm2 HOOK.

If I upgrade to the latest packages, add the lvm2 HOOK, rebuild the initramfs and reboot the system comes up successfully. For clarity, my working HOOKS array with 2.02.103-1 is:

HOOKS="base udev autodetect modconf block mdadm_udev lvm2 filesystems keyboard fsck keymap"

I'm still none the wiser as to why this is now required - none of the LVM partitions are required for the boot process and they used to be mounted once the initramfs had handed off to the second-stage boot. Now the initramfs stage does not complete without the LVM partitions enumerated.

Still, a fix is a fix, and I'll mark this thread as solved; it might save somebody else tearing their hair out!

romsom

Member

Registered: 2012-08-27

Posts: 4

Re: [SOLVED] LVM2/device-mapper upgrade 2.02.103-1 renders PC unbootable

Adding the lvm2 hook didn't help me.
To clarify: My system doesn't use lvm for boot or root, but only for data partitions, which are not discovered unless I run

# vgchange -ay

manually.

#lvdisplay

however lists my volumes anyway.
Modifying mkinitcpi.conf like it was mentioned above and recreating the initramdisk does NOT change anything about this behaviour. (EDIT: at least for me!)

Regards, Roman

Last edited by romsom (2013-10-18 20:06:50)

DcUK

Member

Registered: 2009-04-09

Posts: 10

Re: [SOLVED] LVM2/device-mapper upgrade 2.02.103-1 renders PC unbootable

Firstly, sorry for resurrecting an ageing thread!

After a system update (the first for around two weeks) my system failed to boot again with very similar symptoms to the above. This time, however, booting a rescue disk and rebuilding the initramfs in a chroot did not fix the problem. The system would hang whilst looking for the (non-root) LVM2 partitions.

Further investigation showed the same root cause as before - the underlying RAID1 array forming the LVM2 PV had not been assembled.

At this point my HOOKS looked like this:

HOOKS="base udev autodetect modconf block mdadm_udev lvm2 filesystems keyboard fsck keymap"

At this point I was a little stuck, the rescue shell I was dropped into would not let me log in. Or rather, it would accept the root password and give me a prompt. However, any command typed at the prompt produced a message saying I wasn't logged in and presented another password prompt.

I tried changing back to the systemd HOOK (and rebuilding the initramfs) in the hope that it would fix the issue:

HOOKS="base systemd autodetect modconf block mdadm_udev lvm2 filesystems keyboard fsck keymap"

This failed in exactly the same way, same error about failing to meet dependency for local filesystems target, same rescue shell that doesn't work. From a rescue chroot I read through the help entries for all the HOOKS and noticed that the mdadm_udev tag says it requires the udev tag in order to work. The help for the systemd tag says it replaces the udev tag - but what if it doesn't? Or at least, what if it does so in a slightly different way? I decided to change the HOOKS line again (and rebuild):

HOOKS="base systemd autodetect modconf block mdadm lvm2 filesystems keyboard fsck keymap"

Success! Reverting back to the legacy mdadm tag in combination with systemd booted successfully.

As a final test I decided to see if my original HOOKS would work with just the mdadm_udev to mdadm change:

HOOKS="base udev autodetect modconf block mdadm lvm2 filesystems keyboard fsck keymap"

This did NOT boot and failed in the same way as the original attempt.

To summarise: I can now only boot successfully (as far as I can see) with the systemd and mdadm HOOKS, all combinations using the udev HOOK no longer work for my setup.

Hopefully this might save somebody else going through the same painful process as me (twice).

WonderWoofy

Member

From: Los Gatos, CA

Registered: 2012-05-19

Posts: 8,414

Re: [SOLVED] LVM2/device-mapper upgrade 2.02.103-1 renders PC unbootable

The lvm2 hook is not inteded to work with the systemd hook.  That is why there are systemd specific hooks not included with those packages.  If you want to use the systemd hook, you should also be using the sd-lvm2 hook.  The mdadm_udev hook on the other hand will work fine in both cases (as long as systemd or udev is included).

DcUK

Member

Registered: 2009-04-09

Posts: 10

Re: [SOLVED] LVM2/device-mapper upgrade 2.02.103-1 renders PC unbootable

Well, in for a penny, in for a pound

Decided to play with this a bit more following that advice and amended my HOOKS as follows and rebuilt:

HOOKS="base systemd autodetect modconf block mdadm_udev sd-lvm2 filesystems keyboard fsck keymap"

This should satisfy all the prerequisites for each module, however it does NOT boot my system. The system hangs in the same place as before attempting to mount the LVM partitions. Prior to this there are no error messages that I can see in the boot output or the logs. It doesn't appear to think it has failed to assemble the RAID1 array, rather it doesn't seem to have even attempted it.

Changing back to the legacy mdadm HOOK and rebuilding:

HOOKS="base systemd autodetect modconf block mdadm sd-lvm2 filesystems keyboard fsck keymap"

This DOES correctly boot my system.