You are not logged in.

#1 2013-02-13 02:47:12

zdevex
Member
Registered: 2012-08-17
Posts: 24

Changes to LVM2 and udev break LVM2 on LUKS?

Hey all,

After today's update I have an error on boot that I can't seem to figure out. I like many people, got an error when building mkinitcpio during the update. I reran mkinitcpio after the update, there were no errors, everything was great.

Before I explain the issue fully, here is my setup.

I have a lvm on a luks partition. On the lvm resides "/" "/home" and "swap". My mkinitcpio HOOKs line looks like this

HOOKS="base udev autodetect modconf block mdadm_udev encrypt lvm2 filesystems keyboard fsck"

Note that as you might have guessed from "mdadm_udev" I am using a raid, but this array is separate from my critical partitions, such root and home.

Now for my issue.

When I boot the machine, it executes up until the encrypt hook. It prompts for my password to decrypt the volume, which if given correctly, it does nothing for an unusually long time, then it prints

end_request: I/O error, dev fd0, sector 0

I don't have a floppy drive.

I can mount the system from installation media and chroot into the system, and everything seems fine. I checked my lvm.conf file to make sure it included the line "use_lvmetad = 1" which it does. I reran,

mkinitcpio -p linux

This did not solve the issue.

Any thoughts on what could be causing this?


Thanks,
-D


############## UPDATE ###############

Working fix here: https://bbs.archlinux.org/viewtopic.php … 2#p1232092

Bug report here: https://bugs.archlinux.org/task/33851

Last edited by jasonwryan (2013-02-14 19:45:56)

Offline

#2 2013-02-13 03:10:20

zdevex
Member
Registered: 2012-08-17
Posts: 24

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Update

I looks like the changes to lvm2 and udev, outlined more fully here http://www.mail-archive.com/arch-dev-pu … 20357.html , are causing lvm2 to be run immediately after udev, at least that is what the console seems to print out at boot. This is despite the fact that my HOOKS specifically request lvm2 to run AFTER encrypt, which is required for lvm2 on luks, to my knowledge.

I don't know how to fix this...I will play with a few things, but it seems that the new update which makes lvms hot pluggable via udev (which is surely great!) makes them totally broken from LVM on LUKS. sad

Any thoughts? Ideas for solution? Am I wrong about this (Please say yes.)?

Last edited by zdevex (2013-02-13 03:11:04)

Offline

#3 2013-02-13 03:17:40

zdevex
Member
Registered: 2012-08-17
Posts: 24

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Update.

It appears that I was correct.

http://www.mail-archive.com/arch-commit … 12971.html

According to this, lvm2 is now an early hook, meaning I would need to either rewrite the encrypt hook so that it is an early hook, or rewrite lvm2 so that it is not.

Or at least so it seems...

Offline

#4 2013-02-13 03:20:09

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

I have the same setup as you on my machines and haven't encountered this error (2 out of 3 rebooted fine so far)...


# edit: on my work laptop, just rebooted:

HOOKS="base udev autodetect block keymap encrypt lvm2 keyboard consolefont fsck shutdown"

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#5 2013-02-13 03:23:42

zdevex
Member
Registered: 2012-08-17
Posts: 24

Re: Changes to LVM2 and udev break LVM2 on LUKS?

jasonwryan wrote:

I have the same setup as you on my machines and haven't encountered this error (2 out of 3 rebooted fine so far)...

Interesting, any thoughts on what else might be causing it?

Does your HOOKS line have all the same hooks that I do, I assume it probably doesn't have mdadm_udev? Are they in the same order?

Offline

#6 2013-02-13 03:25:16

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

They are in the order I posted them. When I get home, I'll update my desktop which is a Raid machine with mdadm_udev in the line, but my default install on all boxes is LVM on LUKS.


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#7 2013-02-13 03:32:43

zdevex
Member
Registered: 2012-08-17
Posts: 24

Re: Changes to LVM2 and udev break LVM2 on LUKS?

jasonwryan wrote:

They are in the order I posted them. When I get home, I'll update my desktop which is a Raid machine with mdadm_udev in the line, but my default install on all boxes is LVM on LUKS.

That is my default install as well.

I just downgraded lvm2 and device-mapper and it solved the problem on my desktop, so it has to be somehow related to those packages. I don't think I am doing anything non-standard on my system, in regards to LUKS and LVM at least.

I haven't updated my laptop, which also has said configuration, without the raid. I will do so now and post the results.

Thanks for the help

-D

Last edited by zdevex (2013-02-13 03:35:16)

Offline

#8 2013-02-13 03:49:18

zdevex
Member
Registered: 2012-08-17
Posts: 24

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Update.

I did not get the error on my laptop. I wonder if it somehow has to do with the fact that I have an LVM on LUKS (different LVM and LUKS than for system critical partitions) on my raid as well.

Any thoughts?

Offline

#9 2013-02-13 04:10:25

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

I would try reupgrading lvm2 and device-mapper and see if it was a one-off gremlin.


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#10 2013-02-13 05:55:54

zdevex
Member
Registered: 2012-08-17
Posts: 24

Re: Changes to LVM2 and udev break LVM2 on LUKS?

jasonwryan wrote:

I would try reupgrading lvm2 and device-mapper and see if it was a one-off gremlin.

Reupgrading brings the issue back.

I am stumped. I will have to start playing with the mdadm_udev HOOK and see if that is causing this.

-D

Offline

#11 2013-02-13 06:12:38

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

I just rebooted my desktop after updating: no issues, with these hooks:

HOOKS="base udev v86d autodetect block mdadm_udev keymap encrypt lvm2 keyboard consolefont fsck shutdown"

You should add shutdown to your hooks line.

I also have the following modules:

MODULES="ext4 dm_mod dm_crypt aes_x86_64 raid1"

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#12 2013-02-13 07:29:39

brachyonic
Member
Registered: 2013-02-13
Posts: 6

Re: Changes to LVM2 and udev break LVM2 on LUKS?

From what I was able to gather in the fallback console, the device nodes for my LVM partitions were never created.
After much stumbling about I was able to bring these up and continue the boot process via "lvm vgmknodes" followed by "udevadm trigger"
As far as I can tell my mkinitcpio.conf and lvm/lvm.conf are both correct and in line with changes outlined in the latest news post.
I did have to mkinitcpio after upgrade because of an error over the path of lvmetad, but the manual mkinitcpio showed no further problems.

Offline

#13 2013-02-13 07:56:31

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#14 2013-02-13 08:13:41

brachyonic
Member
Registered: 2013-02-13
Posts: 6

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Possibly, I do get fd0 related errors and don't have a floppy drive, only difference I see is I'm not using encryption on the disk.

Offline

#15 2013-02-13 08:17:44

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Maybe post some more details of your setup so that any similarities can be correlated...


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#16 2013-02-13 08:50:55

tomk
Forum Fellow
From: Ireland
Registered: 2004-07-21
Posts: 9,839

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Offline

#17 2013-02-13 11:02:09

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 671
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

@tomk, maybe he saw it

brachyonic wrote:

As far as I can tell my mkinitcpio.conf and lvm/lvm.conf are both correct and in line with changes outlined in the latest news post.
I did have to mkinitcpio after upgrade because of an error over the path of lvmetad, but the manual mkinitcpio showed no further problems.

Btw yesterday change isn't that new:
[arch-general] Big changes to LVM2 in testing (Wed Oct 31 21:05:30 EDT 2012).
So any user who got Arch installed with lvm2 starting with the November? December? install media (or installed through the net after these change went mainstreamas I did) already has the appropriate lvm.conf file. And should just take care of running 'mkinitcpio -p linux(-FLAVOR)' again after the update to avoid any problems.

Now, looking through lvm.conf I realized that TRIM wasn't activated in lvm, as I forgot to change 'issue_discards' from 0 to 1 til this day. Another benefit from the lvm change news smile

EDIT: typos

Last edited by kozaki (2013-02-13 11:03:07)


Seeded last month: Arch 50 gig, derivatives 1 gig
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop #1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) #2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

#18 2013-02-13 15:45:59

louis058
Member
Registered: 2010-11-12
Posts: 31

Re: Changes to LVM2 and udev break LVM2 on LUKS?

I have set the use_lvmetad option in lvm.conf to 1, I have both udev and lvm2 hooks set in my HOOKS array, and have followed all the instructions given in the news, with one exception: I forgot to rebuild mkinitcpio.
However, afterwards, when fixing this error, I rebuilt mkinitcpio several times.

The error I am getting is during boot, the udev hook will be loaded, then the lvm2 hook will attempt to load, but then the boot process will stop in the middle of that. I have two volume groups, one on an SSD, and one on a HDD, the volume group on the SSD has two logical volumes, and that one is loaded first, then presumably the next step is to load the other volume group on the HDD, but the boot process stops at that point.

My hooks array looks like this:

HOOKS="base udev autodetect modconf block keymap lvm2 filesystems keyboard fsck timestamp"

It almost seemed like the udev hook was loading, then the lvm2 hook was loaded before any of the other hooks before it, so I tried an alternate ordering of hooks.

I tried to place the lvm2 hook directly after udev:

HOOKS="base udev lvm2 autodetect modconf block keymap filesystems keyboard fsck timestamp"

However, this didn't work either.

I also tried downgrading both the lvm2 and the device-mapper packages both to version 2.02.98-1, while changing lvm.conf back, but this didn't work either.

I also found out that I had set the kernel options to load the fallback kernel without the autodetect hook a long time ago, and forgot about it, so it isn't a case of modules either.

Offline

#19 2013-02-13 17:02:32

brain0
Developer
From: Aachen - Germany
Registered: 2005-01-03
Posts: 1,382

Re: Changes to LVM2 and udev break LVM2 on LUKS?

The error I am getting is during boot, the udev hook will be loaded, then the lvm2 hook will attempt to load

The lvm2 hook will not "load" at all. Your description is so obscure, I'll need pictures of your screen (lucky for you, as your system hangs, you have enough time to take some).

Offline

#20 2013-02-13 17:35:20

brachyonic
Member
Registered: 2013-02-13
Posts: 6

Re: Changes to LVM2 and udev break LVM2 on LUKS?

kozaki wrote:

@tomk, maybe he saw it

brachyonic wrote:

As far as I can tell my mkinitcpio.conf and lvm/lvm.conf are both correct and in line with changes outlined in the latest news post.
I did have to mkinitcpio after upgrade because of an error over the path of lvmetad, but the manual mkinitcpio showed no further problems.

Btw yesterday change isn't that new:
[arch-general] Big changes to LVM2 in testing (Wed Oct 31 21:05:30 EDT 2012).
So any user who got Arch installed with lvm2 starting with the November? December? install media (or installed through the net after these change went mainstreamas I did) already has the appropriate lvm.conf file. And should just take care of running 'mkinitcpio -p linux(-FLAVOR)' again after the update to avoid any problems.

Now, looking through lvm.conf I realized that TRIM wasn't activated in lvm, as I forgot to change 'issue_discards' from 0 to 1 til this day. Another benefit from the lvm change news smile

EDIT: typos

Yeah this system was reinstalled less than 1 week ago, so I didn't actually have to do anything. besides the manual mkinitcpio

jasonwryan wrote:

Maybe post some more details of your setup so that any similarities can be correlated...

I'm not sure what all'd be useful information here. My system is pretty basic, things I can see that might be of interest:
GPT partition table with syslinux gptmbr on boot partition, no UEFI
LVM for root swap and home partitions, with home partition spanning two drives

Otherwise I think my setup is fairly vanilla and boring.

Offline

#21 2013-02-13 17:55:19

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

Can you post your HOOKS and MODULES lines from mkinitcpio.conf?


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#22 2013-02-13 18:22:41

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

I got kicked in the face by the LVM update too. For now I have downgraded lvm2 and device-mapper as mentioned in the other thread.

I somehow missed the part about the error in the news item (and the error itself) so I didn't manually rerun mkinitcpio (I did check everything else though). During shutdown the system hung. I only noticed later because the system was still on and the screen was still lit but completely black. I could not access any console output so I have no idea what happened. Now that I'm back in the system I am going to check the logs (scratch that, most of my logs are empty, guess I messed something up when I switched to systemd).

After booting into the recovery partition I was able to mount the system in a chroot and re-run mkinitcpio, which complete without error, but I still get the same error messages at boot (end_request I/O error dev fd0 sector 0).

In mkinitcpio.conf my modules are normally empty but I have tried

MODULES="ext4 dm_mod dm_crypt aes_x86_64"

as I was desperately trying everything.

I have tried adding different hooks (didn't have shutdown in there before) as well as tweaking the order (had lvm2 after block before). The current line is

HOOKS="base udev autodetect block filesystems lvm2 keymap keyboard fsck shutdown"

Partition setup:
1 physical partition for /boot
1 physical partition for lvm
root, var and home are lvm partitions (no encryption)
no raid

Any help would be appreciated. I really don't have the time to fsck around with this right now (yeah, I know, my fault for upgrading, but it normally "just works"™)

Last edited by Xyne (2013-02-13 18:25:58)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#23 2013-02-13 18:28:07

brachyonic
Member
Registered: 2013-02-13
Posts: 6

Re: Changes to LVM2 and udev break LVM2 on LUKS?

jasonwryan wrote:

Can you post your HOOKS and MODULES lines from mkinitcpio.conf?

MODULES=""
HOOKS="base udev autodetect modconf block lvm2 filesystems keyboard fsck"

Offline

#24 2013-02-13 19:39:10

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: Changes to LVM2 and udev break LVM2 on LUKS?

I also use LVM with a configuration similar to Xyne's. I updated this morning and my system booted fine. The only thing unusual about my setup is that I update in chroot from another (minimal) up-to-date Arch system on the same machine. Here's some details, I'm happy to provide anything else if needed.

# fdisk -l /dev/sd?
Disk /dev/sda: 251.1 GB, 251059544064 bytes, 490350672 sectors
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     2099199     1048576   83  Linux
/dev/sda2   *     2099200     2303999      102400   83  Linux
/dev/sda3         2304000   490350671   244023336   8e  Linux LVM

Disk /dev/sdb: 251.1 GB, 251059544064 bytes, 490350672 sectors
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   490350671   245174312   8e  Linux LVM

Disk /dev/sdc: 251.1 GB, 251059544064 bytes, 490350672 sectors
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048   490350671   245174312   8e  Linux LVM

Note: sda1 is the minimal system, sda2 is /boot for both systems. sda3 and sdb1 make up VG0. sdc1 makes up VG1.

# lvscan
  ACTIVE   Original '/dev/VG0/lv_root' [5.00 GiB] inherit
  ACTIVE   Original '/dev/VG0/lv_home' [10.00 GiB] inherit
  ACTIVE            '/dev/VG0/lv_data' [212.00 GiB] inherit
  ACTIVE   Original '/dev/VG0/lv_var' [5.71 GiB] contiguous
  ACTIVE            '/dev/VG0/lv_temp' [10.00 GiB] inherit
  ACTIVE            '/dev/VG0/lv_build' [10.00 GiB] contiguous
  ACTIVE   Snapshot '/dev/VG0/ss_root' [5.00 GiB] inherit
  ACTIVE   Snapshot '/dev/VG0/ss_var' [5.71 GiB] inherit
  ACTIVE   Snapshot '/dev/VG0/ss_home' [10.00 GiB] inherit
  ACTIVE            '/dev/VG1/lv_bucket' [230.00 GiB] inherit
  ACTIVE            '/dev/VG1/lv_swap1' [3.81 GiB] inherit
# cat /etc/mkinitcpio.conf
MODULES="nouveau"
BINARIES=""
FILES=""
HOOKS="timestamp base udev autodetect block lvm2 filesystems fsck shutdown"
COMPRESSION="xz"
COMPRESSION_OPTIONS="-e -9"

I saw something about fd0 when I first ran lvscan after a boot, but it didn't appear again that boot and I haven't tried to reproduce. I also had some errors related to my snapshots during boot:

Feb 13 10:53:18 caddywhompus kernel: device-mapper: table: 254:4: snapshot: Snapshot cow pairing for exception table handover failed
Feb 13 10:53:18 caddywhompus kernel: device-mapper: ioctl: error adding target to table
Feb 13 10:53:18 caddywhompus kernel: device-mapper: table: 254:14: snapshot: Snapshot cow pairing for exception table handover failed
Feb 13 10:53:18 caddywhompus kernel: device-mapper: ioctl: error adding target to table

Note: the first and third lines are "error red"


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#25 2013-02-13 19:39:15

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Changes to LVM2 and udev break LVM2 on LUKS?

The only constant I see in all of the failed systems (at least in the mkinitcpio.confs) is the use of the filesystem hook: perhaps someone could try removing it (and adding the relevant filesystem types to their modules line) and regenerating the intird before rebooting?

# edit: scratch that, alphaniner's post has scuppered that theory...


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

Board footer

Powered by FluxBB