[SOLVED] Boot problems after systemd upgrade

MickeyRat · 2021-05-22 19:22:52

I have a laptop set up to boot UEFI with systemd boot and LVM on Luks. I've been using it this way for a couple years. I just did a system update and now it's failing to boot.

Part of that system update loaded an updated systemd and the folowing in the pacman log indicates that systemd boot was modified.

[2021-05-22T08:01:01-0400] [ALPM-SCRIPTLET] Copied "/usr/lib/systemd/boot/efi/sy
stemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
[2021-05-22T08:01:01-0400] [ALPM-SCRIPTLET] Copied "/usr/lib/systemd/boot/efi/sy
stemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".

That's probably the cause of my problem. I just don't know what to do about it.

When I boot it prompts me for a password and when I provide it I get the following error:

ERROR: device '/dev/mapper/vg1-root' not found. Skipping fsck.
mount: /new_root: no file system specified.

Then it drops me into the failsafe shell. From there, I don't see anything informative in dmesg. In /dev/mapper I can see that the Luks volume has been opened but, none of the LVM volumes are active.

If I boot off of arch install media, I can mount the LVM volumes just fine. Fsck says the filesystems are intact as well. I used arch-chroot to run mkinitcpio. It runs but, it didn't fix anything.

If there's a log file I can use to get more info, please tell me!

The only things I can think of that really have a bearing are the HOOKS line in mkinitcpio (the only thing I've ever changed in there) which is:

HOOKS=(base udev autodetect modconf block keyboard keymap encrypt lvm2 filesystems fsck)

The arch.conf loader is as follows.

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options cryptdevice=UUID=a2924e73-58d5-414d-8776-8c59100e7d54:lvm:allow-discards root=/dev/mapper/vg1-root ipv6.disable=1 quiet rw

Any pointers would be appreciated.

Last edited by MickeyRat (2021-05-26 15:46:56)

seth · 2021-05-22 20:39:24

Possibly related: https://bbs.archlinux.org/viewtopic.php?id=266559

MickeyRat · 2021-05-22 20:57:46

seth wrote:

Possibly related: https://bbs.archlinux.org/viewtopic.php?id=266559

Thanks for the reply. Unfortunately. I don't think that's it. It seems to be getting by the dm-crypt part. As near as I can tell it's not loading LVM properly. The encrypted volume does show up in /dev/mapper in the failsafe shell. it's the LVM volumes that aren't making it.

I'll add that in my original post. The snippet I posted from the pacman log was a trigger that I have in there that runs a bootctl update. Since that post, I tried going in with arch-chroot and doing a bootctl update manually. There really wasn't any good reason to expect that to do anything and it didn't. I also tried changing the /dev/mapper/vg1-root in arch.conf to the UUID. All that did was change the error message to the UUID.

MickeyRat · 2021-05-23 14:52:03

A little more info. I tried taking the quiet out of the kernel parameters in arch.conf. The only additional information that provided was that it was waiting 10 seconds for the root device. That should have been plenty of time but, I tried putting in a rootdelay of 30 seconds. That just made it take longer to dump me into the failsafe shell with the same error message. I'll wait and hope for a better answer. Then I may try reinstalling the boot loader or switching to grub.

MickeyRat · 2021-05-24 12:58:31

I think I've been looking at this wrong. I'll admit I'm no expert in this area but, as far as I know, the boot loader's main job is to fire up the kernel. That's happening otherwise I wouldn't be getting the password prompt nor would I be seeing the LUKS volume in /dev/mapper. The kernel was upgraded during that system upgrade as well and I'm now thinking that's the problem. This thread in the kernel section is talking about something similar to what I'm seeing.

I'll continue to monitor this thread and if I find a solution, I'll post back.

MickeyRat · 2021-05-24 20:06:10

Well after a LOT of digging I decided to downgrade the kernel. I was able to do that successfully but, it didn't fix the problem. So, I tried to downgrade systemd. I wasn't able to do that successfully. mkinitcpio didn't like it at all. So, I've got a hosed system and I'll be doing a reinstall. Can't really call this thread solved but, it's done.

Slithery · 2021-05-24 21:09:42

MickeyRat wrote:

So, I've got a hosed system and I'll be doing a reinstall. Can't really call this thread solved but, it's done.

No need to reinstall, just boot from the Installation media and roll the entire system back to a working date using the archive.

MickeyRat · 2021-05-24 23:44:39

Slithery wrote:

No need to reinstall, just boot from the Installation media and roll the entire system back to a working date using the archive.

Actually I probably don't need to use the archive. This isn't my main system. It's one I use for travel etc. That's why it's encrypted. It's also why I've been willing to deal with it this long. Unless I'm using it a lot, I usually update it once a week and I keep 3 versions of everything that gets updated in the pacman cache. I really wanted to determine what the problem was and I learned a bit in the effort even though I didn't get there. I can probably come up with a find command that will tell me what got updated Saturday (when this happened) and reinstall the previous versions of all that. If I get to a system that can boot, I can update things one at a time to figure out what did it.

I actually have another option as well. Every time I update the system, I use backintime to take a backup. That said, my experience with backintime so far says that it's good for the situation when you screw up and blow away something you should have kept but, for whole partitions it's not that great. OTOH the last time I tried that was some time ago. I may try to come up with a some kind of way to do that. Typically I use the graphical interface but there is a command line interface. Not sure how useful it is.

Damn here I am thinking I'm done and you go and talk me into continuing to try! That's a joke. Thanks for the reply

MickeyRat · 2021-05-25 16:08:10

Here's where I'm at. I've downgraded everything to the point before I had the problem. I reinstalled the systemd boot loader as well. The problem's still there.

One thing I need to note because I haven't seen it mentioned anywhere. Before you use pacman -U to install an older kernel, you need to rename /boot/vmlinuz-linux. If you don't, it won't get replaced and mkinitcpio won't work. It took me a bit to figure that out.

I could try switching to grub but, at this point, I really don't think that will do it. It looks like the boot loader's working far enough to load the kernel. So, unless I get a better suggestion, I'm down to pulling things out of my backintime backup or reinstalling.

I have some other things to do. I'll make up my mind later.

CarbonChauvinist · 2021-05-26 10:59:09

You could perhaps try switching to systemd init?

MickeyRat wrote:

... root=/dev/mapper/vg1-root ...
...
ERROR: device '/dev/mapper/vg1-root' not found. Skipping fsck.
mount: /new_root: no file system specified.

Maybe also try referencing the UUID for your root in boot options instead of the dev mapper name?

MickeyRat · 2021-05-26 15:46:26

CarbonChauvinist wrote:

You could perhaps try switching to systemd init?
MickeyRat wrote:
... root=/dev/mapper/vg1-root ...
...
ERROR: device '/dev/mapper/vg1-root' not found. Skipping fsck.
mount: /new_root: no file system specified.
Maybe also try referencing the UUID for your root in boot options instead of the dev mapper name?

I didn't mention it but, I did try that. It just changed the error message to the UUID.

I'm disappointed that I wasn't able to determine how to fix this but, since I'd tried just about everything, I decided to do a reinstall. It's on an SSD so I backed everything up to an external drive and took it to bare bones with a secure erase then restored the home volume from the external drive. I still have a little configuring to do but, it's booting and I have a desktop.

A little advice for those that do this. Take a good look at this wiki and save your installed packages. You can use those lists to install everything quickly.

Thanks to all those that viewed this thread.

Last edited by MickeyRat (2021-05-26 15:47:29)

Arch Linux

#1 2021-05-22 19:22:52

[SOLVED] Boot problems after systemd upgrade

#2 2021-05-22 20:39:24

Re: [SOLVED] Boot problems after systemd upgrade

#3 2021-05-22 20:57:46

Re: [SOLVED] Boot problems after systemd upgrade

#4 2021-05-23 14:52:03

Re: [SOLVED] Boot problems after systemd upgrade

#5 2021-05-24 12:58:31

Re: [SOLVED] Boot problems after systemd upgrade

#6 2021-05-24 20:06:10

Re: [SOLVED] Boot problems after systemd upgrade

#7 2021-05-24 21:09:42

Re: [SOLVED] Boot problems after systemd upgrade

#8 2021-05-24 23:44:39

Re: [SOLVED] Boot problems after systemd upgrade

#9 2021-05-25 16:08:10

Re: [SOLVED] Boot problems after systemd upgrade

#10 2021-05-26 10:59:09

Re: [SOLVED] Boot problems after systemd upgrade

#11 2021-05-26 15:46:26

Re: [SOLVED] Boot problems after systemd upgrade

Board footer