Changes to LVM2 and udev break LVM2 on LUKS?

brain0 · 2013-02-13 19:43:00

alphaniner wrote:

I saw something about fd0 when I first ran lvscan after a boot, but it didn't appear again that boot and I haven't tried to reproduce. I also had some errors related to my snapshots during boot:
Feb 13 10:53:18 caddywhompus kernel: device-mapper: table: 254:4: snapshot: Snapshot cow pairing for exception table handover failed
Feb 13 10:53:18 caddywhompus kernel: device-mapper: ioctl: error adding target to table
Feb 13 10:53:18 caddywhompus kernel: device-mapper: table: 254:14: snapshot: Snapshot cow pairing for exception table handover failed
Feb 13 10:53:18 caddywhompus kernel: device-mapper: ioctl: error adding target to table
Note: the first and third lines are "error red"

These messages are bad, are your snapshots working? What does systemctl status lvm-monitoring.service say?

alphaniner · 2013-02-13 19:57:30

Immediately after boot I did lvdisplay for the snapshots, and everything seemed fine:

# lvdisplay VG0/ss_root
  --- Logical volume ---
  LV Path                /dev/VG0/ss_root
  LV Name                ss_root
  VG Name                VG0
  LV UUID                2TJJwf-d4H1-sKiZ-Offv-eIdE-325x-NLceQK
  LV Write Access        read/write
  LV Creation host, time recover, 2013-02-13 10:45:53 -0500
  LV snapshot status     active destination for lv_root
  LV Status              available
  # open                 0
  LV Size                5.00 GiB
  Current LE             1280
  COW-table size         5.00 GiB
  COW-table LE           1280
  Allocated to snapshot  5.86%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4

So I didn't worry about it. But

#  systemctl status lvm-monitoring.service
lvm-monitoring.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
	  Loaded: loaded (/usr/lib/systemd/system/lvm-monitoring.service; disabled)
	  Active: inactive (dead)
	    Docs: man:dmeventd(8)
	          man:lvcreate(8)
	          man:lvchange(8)
	          man:vgchange(8)

Then, on the other hand:

# journalctl -b
...
Feb 13 10:53:21 caddywhompus lvm[495]: Monitoring snapshot VG0-ss_root
Feb 13 10:53:21 caddywhompus lvm[495]: Monitoring snapshot VG0-ss_home
Feb 13 10:53:21 caddywhompus lvm[495]: Monitoring snapshot VG0-ss_var
...

Edit: Fixed the lvdisplay output.

Anyway, I'm not too worried about this for the time being, so let's not hijack the thread from the people who can't even boot.

Last edited by alphaniner (2013-02-13 20:06:14)

kozaki · 2013-02-13 20:34:54

Everything went fine in this location after today's update.
system has a more than simple lvm setup: 2 VGs, one on the SSD (to allow for quick and easy resizing just in case), and one the HDD (so that another one can be added in the working partition layout when it comes )

mkinitcpio.conf:

MODULES="ahci libahci ehci-hcd sd_mod ext2 ext4 vfat"
BINARIES="fsck fsck.ext4 fsck.ext2 fsck.vfat"
HOOKS="base udev modconf lvm2 keymap timestamp"

lvm.conf:

 # If lvmetad has been running while use_lvmetad was 0, it MUST be stopped
 # before changing use_lvmetad to 1 and started again afterwards.
 use_lvmetad = 1

systemctl status lvm-monitoring.service

lvm-monitoring.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
<SNIP>
févr. 13 21:18:48 llewellyn lvm[11661]: 6 logical volume(s) in volume group "vg_ssd" monitored
févr. 13 21:18:48 llewellyn lvm[11661]: 2 logical volume(s) in volume group "data_vg" monitored

kernel Command line:

BOOT_IMAGE=/vmlinuz-linux-ck root=/dev/mapper/vg_ssd-lv_linux1 ro ipv6.disable=1 libahci.ignore_sss=1 pcie_aspm=force quiet

When I generated the self compiled kernel today I got a warning about (lvm snapshot module) --that I don't use atm. Errr sorry I can't find it in the journal, must get used with journalctl (the sooner the better).

EDIT: systemd-analyze blame shows lvmetad:

 23ms lvmetad.service

Good luck

Last edited by kozaki (2013-02-13 21:14:55)

maveloth · 2013-02-13 21:20:42

Hi this evening I updated my machines and after that one can boot fine(my notebook NB in the configuration details) and one(I'll call it PC in the configuration details) can't boot with this errors:

...
mdadm /dev/md/system has been started with 2 devices
device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
device-mapper: table: 253:2: linear: dm-linear: Device lookup failed
device-mapper: table: 253:3: linear: dm-linear: Device lookup failed
device-mapper: table: 253:4: linear: dm-linear: Device lookup failed
mdadm /dev/md/data has been started with 2 devices
device-mapper: table: 253:5: linear: dm-linear: Device lookup failed
device-mapper: table: 253:5: linear: dm-linear: Device lookup failed
device-mapper: table: 253:5: linear: dm-linear: Device lookup failed
ERROR: device '/dev/mapper/ArchVol-lvroot' not found. Skipping fsck
ERROR:unable to find root devices '/dev/mapper/ArchVol-lvroot'.
...

and i was prompted to the recovery shell.

PC:
kernel command line:

linux /vmlinuz-linux-ck root=/dev/mapper/ArchVol-lvroot ro lvmwait=/dev/md/system quiet

mkinitcpio.conf:

MODULES="raid1 dm-mod"
HOOKS="base udev autodetect modconf block mdadm lvm2 filesystems keyboard fsck"

lsblk(I had 2 mdadm array RAID1 using 2 pair of disks so I omitted 2 drives below because are identical):

sdb                               8:16   0 931,5G  0 disk  
├─sdb1                            8:17   0     2M  0 part  
└─sdb2                            8:18   0 931,5G  0 part  
  └─md127                         9:127  0 931,4G  0 raid1 
    ├─ArchVol-lvboot (dm-0)     253:0    0   300M  0 lvm   /boot
    ├─ArchVol-lvroot (dm-1)     253:1    0    10G  0 lvm   /
    ├─ArchVol-lvhome (dm-2)     253:2    0   200G  0 lvm   /home
    ├─ArchVol-lvvar (dm-3)      253:3    0    20G  0 lvm   /var
    └─ArchVol-lvswap (dm-4)     253:4    0     4G  0 lvm   SWAP
sdc                               8:32   0   1,8T  0 disk  
└─sdc1                            8:33   0   1,8T  0 part  
  └─md126                         9:126  0   1,8T  0 raid1 
    └─volData-multimedia (dm-5) 253:5    0   1,5T  0 lvm   /data

NB:
kernel command line:

linux   /vmlinuz-linux-ck root=/dev/mapper/vg_maveloth--nb-lv ro  resume=/dev/sda6 lvmwait=/dev/sda5 quiet

mkinitcpio.conf:

MODULES="dm_mod"
HOOKS="base udev lvm2 autodetect modconf block filesystems keyboard fsck"

lsblk:

NAME                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                     8:0    0 465,8G  0 disk 
├─sda1                                  8:1    0   100M  0 part 
├─sda2                                  8:2    0 119,9G  0 part 
├─sda3                                  8:3    0   500M  0 part /boot
├─sda4                                  8:4    0     1K  0 part 
├─sda5                                  8:5    0   200G  0 part 
│ ├─vg_maveloth--nb-lvhome (dm-0)     254:0    0    95G  0 lvm  /home
│ ├─vg_maveloth--nb-lvvar (dm-1)      254:1    0    15G  0 lvm  /var
│ ├─vg_maveloth--nb-lv (dm-2)         254:2    0    15G  0 lvm  /
│ ├─vg_maveloth--nb-lvarchroot (dm-3) 254:3    0    15G  0 lvm  
│ ├─vg_maveloth--nb-lvarchvar (dm-4)  254:4    0    15G  0 lvm  
│ └─vg_maveloth--nb-lvvirt (dm-5)     254:5    0    30G  0 lvm  /virt
├─sda6                                  8:6    0     4G  0 part SWAP
└─sda7                                  8:7    0   500M  0 part

On both my machine I rebuilt the initram after the update because I noticed the error in the output( for both the kernel I had linux and linux-ck but both can't boot). My lvm.conf are pretty new so I didn't had to modify them "lvmetad=1" was already set.
I tried to chroot my box and rebuild again my initram but no results, and I had also downgraded lvm2 and device-mapper (via pacman -U from the cache dir) and still prompted to recovery shell, so I check the folder /dev/mapper and it was not populated, so tried to issue "vgscan && vgchange -ay" then exiting the shell PC had booted.

Sorry for my long post I hope I had provided all the usefull information. I'm Italian so I'm also sorry for my bad english

Maveloth

EDIT: I forget to mention that PC had the 4 disks with GPT partition table but NB had an MBR partition table.

Last edited by maveloth (2013-02-13 22:02:56)

alphaniner · 2013-02-13 23:05:55

maveloth wrote:

EDIT: I forget to mention that PC had the 4 disks with GPT partition table but NB had an MBR partition table.

So, non-working system is GPT and working system is MBR. My disks are all MBR as well. Could that be the common characteristic?

jasonwryan · 2013-02-13 23:07:55

No: I have a mix of MBR and GPT...

Xyne · 2013-02-13 23:22:43

I noticed that I didn't have the lvm service enabled, so I enabled it, re-upgraded, rebuilt the ramdisk, and rebooted... no luck.

Incidentally, can someone confirm that this is normal in lvm2 2.02.98-1?

 systemctl status lvm-monitoring.service
lvm-monitoring.service
	  Loaded: error (Reason: No such file or directory)
	  Active: inactive (dead)

digirium · 2013-02-13 23:29:02

@Xyne status output looks like this my system:

lvm-monitoring.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
	  Loaded: loaded (/usr/lib/systemd/system/lvm-monitoring.service; enabled)
	  Active: active (exited) since Wed 2013-02-13 22:40:30 GMT; 47min ago
	    Docs: man:dmeventd(8)
	          man:lvcreate(8)
	          man:lvchange(8)
	          man:vgchange(8)
	 Process: 147 ExecStart=/usr/sbin/lvm vgchange --monitor y (code=exited, status=0/SUCCESS)

hellknight · 2013-02-13 23:40:08

He means that the boot process stops at "::running early hook [lvm2]".
I have the same problem and I also forgot the same thing, rebuild the kernel image with mkinitcpio, which I did afterwards from a chrooted environment. My logical volumes are also in a SSD by the way.

Last edited by hellknight (2013-02-13 23:43:04)

Xyne · 2013-02-13 23:58:33

@digirium
For lvm2 2.02.98-1?

jasonwryan · 2013-02-14 00:00:02

I don't use it:

┌─[Veles ~]
└─╼ systemctl status lvm-monitoring.service
lvm-monitoring.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
          Loaded: loaded (/usr/lib/systemd/system/lvm-monitoring.service; disabled)
          Active: inactive (dead)
            Docs: man:dmeventd(8)
                  man:lvcreate(8)
                  man:lvchange(8)
                  man:vgchange(8)

Xyne · 2013-02-14 00:27:08

Sorry, I should have been clearer. I was asking if it's normal that I do not have the service file itself (look at my output). Was the service added in the last update?

digirium · 2013-02-14 00:31:18

Xyne wrote:

@digirium
For lvm2 2.02.98-1?

pacman -Q | fgrep lvm2 = lvm2 2.02.98-3

Edit: Sorry, reading failure my bad.

Last edited by digirium (2013-02-14 00:32:29)

jasonwryan · 2013-02-14 00:32:52

No, it isn't normal: I have had the lvm service file since switching these machines over to systemd (I remember debating whether it was necessary to enable it or not).

Your lvmetad service is running though?

maveloth · 2013-02-14 00:33:43

For lvm2 2.02.98-1?

No this service file si provided by the new version of lvm pkg and the lvm.service is no more present.

[root@mav-4desk maveloth]# tar -tvf /var/cache/pacman/pkg/lvm2-2.02.98-1-x86_64.pkg.tar.xz |grep service
-rw-r--r-- root/root       319 2012-10-15 23:21 usr/lib/systemd/system/lvm.service
-rw-r--r-- root/root       345 2012-10-15 23:21 usr/lib/systemd/system/lvm-on-crypt.service
[root@mav-4desk maveloth]# tar -tvf /var/cache/pacman/pkg/lvm2-2.02.98-3-x86_64.pkg.tar.xz |grep service
-rw-r--r-- root/root       566 2012-11-03 19:21 usr/lib/systemd/system/lvm-monitoring.service
-rw-r--r-- root/root       334 2012-11-03 19:21 usr/lib/systemd/system/lvmetad.service
[root@mav-4desk maveloth]#

I tried to insert the filesystem modules in the initram but for me was useless, still can't boot. I also tried to use the "auto_activation_volume_list" and still no results. At the end I tried to revert the above changes to the initram and use the "mdadm_udev" hook instad of "mdadm" and now I can boot.

And this is my kernel command line:

linux   /vmlinuz-linux-ck root=/dev/mapper/ArchVol-lvroot ro  dolvm quiet

Hope this can help someonelse.

Maveloth

P.S. I also deleted the link /etc/systemd/system/basic.target.wants/lvm.service because with the latest lvm pkg this service doesn't exist.

Last edited by maveloth (2013-02-14 00:36:32)

cfr · 2013-02-14 01:40:35

For data, I have LVM on LUKS. Although my laptop currently has serious problems, this is one thing which seems to work:

MODULES="i915"
HOOKS="base udev autodetect modconf block encrypt lvm2 resume filesystems keyboard fsck timestamp shutdown"

Everything bar /boot and /boot/efi is encrypted. kernel command line:

Command line: root=/dev/vgroup-cfr/arch ro rootfstype=ext4 cryptdevice=/dev/disk/by-uuid/XXX-XXX-XXX-XXX-XXX:lvm resume=/dev/vgroup-cfr/swap pcie_aspm=force i915.i915_enable_rc6=-1 i915.i915_enable_fbc=1 i915.lvds_downclock=1 i915.semaphores=1 add_efi_memmap initrd=initramfs-linux.img

Xyne · 2013-02-14 01:49:12

jasonwryan wrote:

Your lvmetad service is running though?

My system isn't even running, so I don't know how to check if the service is started when it tries to boot.

I didn't have the dolvm parameter on the kernel line. I'm trying again with that.

edit: nope, didn't change anything, still fails to boot

Last edited by Xyne (2013-02-14 02:06:14)

Xyne · 2013-02-14 02:15:10

Opened a ticket for this: https://bugs.archlinux.org/task/33851

lukas2511 · 2013-02-14 02:44:42

On my system the initram scripts do not start the volume group, everything else seems to be working.

As my system drops to the shell, I have to run "lvm lvchange -a e vg0" followed by "exit" and it continues booting.

Problem is that I do not have access to the hardware on all of my systems... so yea... downgraded the three packages (lvm2, device-mapper, linux) and system is booting again.

luciferin · 2013-02-14 04:45:50

Make sure you disable your floppy drive in your system BIOS. LVM2 seems to lock up while searching for an LVM on the floppy drive that isn't there. Maybe using the auto_activation_volume_list option in /etc/lvm/lvm.conf to pick specific drives to activate would also remedy this problem?

I just made this change and it fixed the error on my system.

jasonwryan · 2013-02-14 05:11:03

Merging with the other threads so all the information is in one place.

First post updated with a reported fix and the bug report.

tranqil · 2013-02-14 08:28:55

I copied my last post from the other (wrong) thread into this one, hope this is ok:

Hello, I have an problem similar to this one. But my boot setup is slightly different, because I use my own mkinitcpio hook to decrypt my disk from smartcard.
I have two GPT-partitions on the harddisk (boot and a lvm "physical volume"), the smartcard hook only LUKS-decrypts the pv and normally without the last changes to udev and lvm in the boot procedure, everything is working and my system boots perfect.
But now I got dropped in the initrd rescue shell, after entering my passphrase to access the plugged in smartcard which decrypts my disk. The strange part is, if I enter "exit" to continue the boot process, everything is working as expected.

The order of my hooks in mkinitcpio.conf are the following: HOOKS="base udev autodetect block keymap smartcard encrypt lvm2 resume filesystems keyboard fsck"
(The "encrypt" hook is only needed to add corresponding binaries to the initrd)

Is the lvm hook really switched to early run in the new version? Maybe this explain my problem, becaus it is running bevore my 'smartcard' hook decrypts the pv.
The main problem is, that I can't switch the "smartcard" hook easily to 'early_run' because it depends on other later hooks (to detect the smartcard reader).

Edit: I'm sorry to say, that luciferin solution does not solve my problem. I have no floppy drive that I can disable in BIOS, and commenting out the auto_activation list brought no changes too.
Here is a picture of the early boot procedure: http://t0sp.net/~mb/images/lvm_small.png | http://t0sp.net/~mb/images/lvm_big.png

Fixed typo: encrypt = decrypt.

Last edited by tranqil (2013-02-14 10:01:34)

brain0 · 2013-02-14 09:41:47

@tranqil: Do not write anything into auto_activation_volume_list unless you know what you are doing (you don't btw, as your entries are wrong, thus you disabled autoactivation entirely). Leave it COMMENTED!

@maveloth: 'device-mapper: table: 253:0: linear: dm-linear: Device lookup failed' - I saw those before by from someone using the mdadm hook - I don't know what they mean, and I don't get those when using the mdadm_udev hook. (There may be an error in the mdadm hook, it misses the mdadm binary which the udev rules need - using the mdadm_udev hook is always a better idea if it works.)

@everyone who gets dropped in a recovery shell: check udevadm info --query=all /dev/your_physical_volume and look if ID_FS_TYPE is set. Also quickly check using 'ps' what processes are running.

@everyone with raid: Compare the results of the mdadm and mdadm_udev hook.

@everyone that hangs during "running early hook lvm2": I have no idea, the daemon doesn't actually do anything before it forks, so this shouldn't happen. I have no clue at all. Maybe send me a copy of your broken initramfs-linux.img.

There's also been a hint that the system may hang when there's a non-existent floppy drive that has been enabled in BIOS (which makes no sense, as scanning is initiated by udev only on devices that are actually LVM members). While this is weird, I would like everyone to check for that. For some reason, the floppy module is added to initramfs by the 'block' hook. Blacklisting the floppy module the kernel command line option modprobe.blacklist=floppy.

As it stands now, I am unable to reproduce any of these problems - all setups I tested boot fine, those include RAID, LUKS and plain LVM setups, and even a setup where I stack LVM on top of LVM (which is insane, but still works without flaws). Dave also tested a setup where LVM stack on top of LUKS, which in turn is inside LVM. None of us is able to break LVM so they could investigate what the problem is. Sorry.

When you get dropped in a recovery shell, you should always be able to run 'lvm vgchance -ay' to activate the volumes and boot - while that shouldn't be necessary, it is still a good workaround to get your systems to boot without too much trouble.

brain0 · 2013-02-14 09:54:13

tranqil wrote:

But now I got dropped in the initrd rescue shell, after entering my passphrase to access the plugged in smartcard which encrypts my disk. The strange part is, if I enter "exit" to continue the boot process, everything is working as expected.

Okay, that's new. Is there no delay before it drops you to the shell? What's your kernel commandline?

tranqil · 2013-02-14 09:59:42

Hello brain0; My kernel command line is: root=/dev/mapper/mojo-root resume=/dev/mapper/mojo-swap loglevel=2 rootdelay=0 ro vga=current quiet The only delay is related in decrypting the disk. This is ok. But after that there is no delay and I dropped instantly into the rescue shell.

@tranqil: Do not write anything into auto_activation_volume_list unless you know what you are doing (you don't btw, as your entries are wrong, thus you disabled autoactivation entirely). Leave it COMMENTED!

I think my entries are correct. It should be enought to add the vg-name "mojo", the lv-names are for backup only. Apart from this I boot my system with this settings.

Last edited by tranqil (2013-02-14 10:02:01)

Arch Linux

#26 2013-02-13 19:43:00

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#27 2013-02-13 19:57:30

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#28 2013-02-13 20:34:54

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#29 2013-02-13 21:20:42

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#30 2013-02-13 23:05:55

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#31 2013-02-13 23:07:55

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#32 2013-02-13 23:22:43

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#33 2013-02-13 23:29:02

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#34 2013-02-13 23:40:08

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#35 2013-02-13 23:58:33

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#36 2013-02-14 00:00:02

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#37 2013-02-14 00:27:08

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#38 2013-02-14 00:31:18

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#39 2013-02-14 00:32:52

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#40 2013-02-14 00:33:43

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#41 2013-02-14 01:40:35

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#42 2013-02-14 01:49:12

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#43 2013-02-14 02:15:10

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#44 2013-02-14 02:44:42

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#45 2013-02-14 04:45:50

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#46 2013-02-14 05:11:03

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#47 2013-02-14 08:28:55

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#48 2013-02-14 09:41:47

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#49 2013-02-14 09:54:13

Re: Changes to LVM2 and udev break LVM2 on LUKS?

#50 2013-02-14 09:59:42

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Board footer