[SOLVED] Volume groups not showing up after update

nunikasi · 2014-07-06 19:45:23

nunikasi wrote:

Hello!
I run a arch linux setup with LVM volumes on a LUKS encrypted drive /dev/sda. The header resides in the initramfs-linux-ck.img (So kernel and header are on a USB stick and my hard drive is completely encrypted).
Everything worked nicely with this setup until today. I ran a
pacman -Syu
and the kernel was updated as normal and made a new initramfs-linux-ck.img on the usb stick (mounted in /boot). This is the way I always updated the kernel, and it always worked, but for some reason, after the reboot, and after typing my passphrase I got an error
ERROR: device '/dev/mapper/nunikasi-root' not found. Skipping fsck.
This was strange, so I booted the latest arch live-cd and did a
cryptsetup luksOpen /dev/sda nunikasi --header (pathToHeaderFile)
and entered the passphrase and still there are no volume groups in /dev/mapper. Am I wrong in thinking that this command should load my volume groups as
/dev/mapper/nunikasi-root
and
/dev/mapper/nunikasi-home
?
Running
vgscan
also reports no volume groups.
Additional information:
Last time I updated the kernel was around 24th of June.
I did run pacman -r linux before the update, assuming there is no need for it since I run linux-ck. I hope this is not relevant.
Do you know what went wrong?

What was the problem?
As I saw that dm-crypt had a problem with finding my logical volumes, I fired up

cryptsetup luksOpen /dev/sda nunikasi --header (pathToHeaderFile)
dhex /dev/mapper/nunikasi

and found that the lvm metadata was cut in half. So I ran

dhex /dev/mapper/nunikasi /dev/sda

and quickly saw that there was an attempt to make a MBR on /dev/sda (I saw the boot signature 55 aa on offset 1FE-1FF). I remembered that I tried to use GParted some days earlier to format a memory stick, and I probably forgot to change from /dev/sda before creating a MBR. This is most probably what happened, and now I had to restore the metadata in some way.

What was the solution?
As I had never made a backup of the metadata, I knew that LVM makes one for you and puts it in /etc/lvm/backup/. So I fired up dhex on /dev/mapper/nunikasi and searched for it. Found the offset and did:

dd if=/dev/mapper/nunikasi count=2148 skip=8928722944 of=lvm-metadata.txt iflag=skip_bytes,count_bytes

Now I had the metadata backup and the right UUID for my physical volume. Only steps left were:

pvcreate -ff -u [UUID] —restorefile lvm-metadata.txt /dev/mapper/nunikasi
vgcfgrestore -f lvm-metadata.txt nunikasi
vgchange -ay

And my setup is working again!

Have a nice day,
nunikasi

Last edited by nunikasi (2014-07-07 20:30:16)

clfarron4 · 2014-07-06 19:52:35

Just as a sanity check, you do have the lvm2 and encrypt/sd-lvm2 and sd-encrypt hooks in /etc/mkinitcpio.conf? (Either the first pair or second pair depending on which set of hooks you use).

Also, can you tell us if there were any other relevant packages updated?

Last edited by clfarron4 (2014-07-06 19:52:56)

nunikasi · 2014-07-06 20:07:41

Thanks for the reply!

clfarron4 wrote:

Just as a sanity check, you do have the lvm2 and encrypt/sd-lvm2 and sd-encrypt hooks in /etc/mkinitcpio.conf? (Either the first pair or second pair depending on which set of hooks you use).

Yes, I have the lvm2 and encrypt hook in /etc/mkinitcpio.conf and they are inside my initramfs-linux-ck.img.

clfarron4 wrote:

Also, can you tell us if there were any other relevant packages updated?

No, I am sad I cannot tell you. I know that the upgrade is relevant, but I just can't see how it is, because my /dev/sda is still a LUKS encrypted disc and opening it with luksOpen should give me the right volume groups under /dev/mapper...

clfarron4 · 2014-07-06 20:28:46

Hmm... Have you tried breaking out an Arch installation disk and having a look with that?

nunikasi · 2014-07-06 20:32:56

clfarron4 wrote:

Hmm... Have you tried breaking out an Arch installation disk and having a look with that?

Yes, I wrote that in the original post.

I remember core/filesystems and some other core packages were updated.

clfarron4 · 2014-07-06 21:34:38

nunikasi wrote:

clfarron4 wrote:
Hmm... Have you tried breaking out an Arch installation disk and having a look with that?
Yes, I wrote that in the original post.

Sorry about that. I've just read it again.

OK, based on the information you've given us (LVM on LUKS, VG=nunikasi and LVs=root,home, backed up header), I would have thought that the commands you issued in your original post would have unlocked it. I'm not going to make any assumptions about the header of the actual LUKS partition though.

nunikasi wrote:

I remember core/filesystems and some other core packages were updated.

I've had a look through my pacman.log and I can't think of anything that might upset it (that said, I am quite tired). I run 5 (maybe 6) different kernels (3.10, 3.12, 3.14 and 3.15 branches) and I haven't had any problems with finding PVs, VGs and LVs on either of my systems.

I'm at a loss as to what could be up right now and I'll this thread for tomorrow (because sleep beckons).

EDIT: Wow. That's quite something.

Last edited by clfarron4 (2014-07-08 11:21:46)

Arch Linux

#1 2014-07-06 19:45:23

[SOLVED] Volume groups not showing up after update

#2 2014-07-06 19:52:35

Re: [SOLVED] Volume groups not showing up after update

#3 2014-07-06 20:07:41

Re: [SOLVED] Volume groups not showing up after update

#4 2014-07-06 20:28:46

Re: [SOLVED] Volume groups not showing up after update

#5 2014-07-06 20:32:56

Re: [SOLVED] Volume groups not showing up after update

#6 2014-07-06 21:34:38

Re: [SOLVED] Volume groups not showing up after update

Board footer