[SOLVED] systemd drops me into emergency shell after upgrade

megthepeng · 2020-09-27 07:36:09

I did a full system upgrade a few weeks ago and then I rebooted, but it only dropped me into an emergency shell. The startup messages said "Failed to start Load Kernel Modules" and "Failed to mount /boot" "Dependency failed for Local File Systems".

This is the output of systemctl status systemctl status systemd-modules-load.service

 ● systemd-modules-load.service - Load Kernel Modules
     Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
     Active: failed (Result: exit-code) since Sun 2020-09-27 12:41:32 EEST; 7min ago
       Docs: man:systemd-modules-load.service(8)
             man:modules-load.d(5)
    Process: 253 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
   Main PID: 253 (code=exited, status=1/FAILURE)

Warning: journal has been rotated since unit was started, output may be incomplete.

and systemctl status boot.mount

● boot.mount - /boot
     Loaded: loaded (/etc/fstab; generated)
     Active: failed (Result: exit-code) since Sun 2020-09-27 12:41:32 EEST; 7min ago
      Where: /boot
       What: /dev/disk/by-uuid/F2BF-E8B2
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)

Sep 27 12:41:32 pizzacubecuri systemd[1]: Mounting /boot...
Sep 27 12:41:32 pizzacubecuri mount[324]: mount: /boot: unknown filesystem type 'vfat'.
Sep 27 12:41:32 pizzacubecuri systemd[1]: boot.mount: Mount process exited, code=exited, status=32/n/a
Sep 27 12:41:32 pizzacubecuri systemd[1]: boot.mount: Failed with result 'exit-code'.
Sep 27 12:41:32 pizzacubecuri systemd[1]: Failed to mount /boot.

and here is the full output of journalctl -xb: https://dpaste.com/27BWF8FKP

I am clueless as to what I should do. Please tell me if you need more information.

Last edited by megthepeng (2020-09-29 18:45:19)

solskog · 2020-09-27 07:51:00

Where: /boot
What: /dev/disk/by-uuid/F2BF-E8B2
mount: /boot: unknown filesystem type 'vfat'.

It's likely your /etc/fstab has miss configured /boot partition with /boot/efi partition.

megthepeng · 2020-09-27 08:19:01

I find it pretty unlikely, as I've been mounting that partition on /boot without problems for months. The only thing that changed was upgrading my kernel from 5.7 to 5.8.

I did try changing the /boot in /etc/fstab to /boot/efi, /boot/EFI and even /efi and it still gave me the same messages but with /boot/efi etc. instead of /boot. When I try mounting it manually I get the same "unknown filesystem type 'vfat' "

solskog · 2020-09-27 08:56:29

It seems system tried to mount EFI/ESP partition on /boot that is not empty.

Sep 27 12:41:32 pizzacubecuri systemd[1]: boot.mount: Directory /boot to mount over is not empty, mounting anyway.
The directory /boot is specified as the mount point (second field in
░░ /etc/fstab or Where= field in systemd unit file) and is not empty.
░░ This does not interfere with mounting, but the pre-exisiting files in
░░ this directory become inaccessible. To see those over-mounted files,
░░ please manually mount the underlying file system to a secondary
░░ location.

I don't see your /etc/fstab file, but the normal way is /boot/ (normally on ext4) contains kernel, initramfs etc.
and /boot/efi/ contains your bootloader file. the Warning says your system tried to mount /dev/disk/by-uuid/F2BF-E8B2 (bootloader in vfat partition) on /boot and failed.
If you tried to repair it through liveUSB, Pls check if vfat module is loaded

#lsmod | grep vfat
# modprobe vfat

Last edited by solskog (2020-09-27 09:27:20)

Daerandin · 2020-09-27 10:03:24

solskog wrote:

I don't see your /etc/fstab file, but the normal way is /boot/ (normally on ext4) contains kernel, initramfs etc.
and /boot/efi/ contains your bootloader file.

I am curious what you base this statement on. Mounting the EFI partition on /boot is not only normal, but recommended if you use certain bootloaders. Only a few bootloaders can boot a kernel that is not located on the EFI partition itself.

As for the boot issue, I'm not expert but to me it sounds like there is a initramfs/kernel mismatch. In particular, this line seems suspicious to me:

 Command line: ro root=UUID=f2e69ac4-645a-45ef-9fa0-9089541a3531 initrd=boot\initramfs-linux.img

What bootloader are you using, and what is the associated configuration file for it? Your fstab will probably also be useful here.

EDIT: Fixed some grammar.

Last edited by Daerandin (2020-09-27 10:04:57)

solskog · 2020-09-27 11:26:23

Daerandin wrote:

Only a few bootloaders can boot a kernel that is not located on the EFI partition itself.

Well, I agree. I just feel much safer with kernel partition and EFI partition separated. In case a damaged EFI partition result lost kernel.

megthepeng · 2020-09-27 13:00:51

Sorry for not mentioning earlier, I use rEFInd as a bootloader and so far it didn't have any problems with the partition being mounted on boot.

Also this is my fstab. I did check the UUIDs with blkid and they match.

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p5
UUID=f2e69ac4-645a-45ef-9fa0-9089541a3531	/         	ext4      	rw,relatime	0 1

# /dev/sda1
UUID=24847fe7-f208-4173-9098-a8bd1528e323	/home     	ext4      	rw,relatime	0 2

# /dev/nvme0n1p1
UUID=F2BF-E8B2      	/boot      	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

My laptop has an SSD with both Win10 and Arch installed and an HDD, which holds /home. I've been using this dual boot layout for quite a while.
Any other ideas on what else I could try?

Daerandin · 2020-09-27 14:26:55

According to your kernel command line, it will look for the initramfs under a directory called 'boot' on your EFI partition. I don't know why it would do that, but I suspect it might be caused by your boot partition not being mounted when you performed the update. I'm not exactly sure how rEFInd handles this as I've never used it myself, but that is what it looks like to me. This is assuming you don't edit the configuration file manually, and also backed by the fact that your system found files under /boot before attempting to mount your EFI partition there.

I would suggest that you boot the live media, ensure everything is mounted properly, the chroot into your system and run a full system upgrade. You might also want to clean up the /boot directory when your EFI partition is not mounted there.

solskog · 2020-09-28 11:37:42

I found this problem interesting and therefore setup a identical boot layout as OP using rEFInd.
When I compared OP and my kernel boot parameters I found differences.

@megthepeng: ro root=UUID=f2e69ac4-645a-45ef-9fa0-9089541a3531 initrd=boot\initramfs-linux.img
@solskog: rw root=UUID=2d534s61-23e-4w25-93e4-9sd46secf834 initrd=\initramfs-linux.img

I noticed that OP's ROOT partition was booted readonly and initramfs was located on a separate /boot partition that is not /dev/nvme0n1p1 (UUID=F2BF-E8B2).
I think the problem could be inside /boot/refind_linux.conf that define the boot parameters. Maybe there are two rebind_linux.conf files and the wrong one was used for his boot.

Last edited by solskog (2020-09-28 21:43:34)

seth · 2020-09-28 16:23:29

Sep 27 12:41:32 pizzacubecuri systemd[1]: Failed to start Load Kernel Modules.

Everything in that log says that the running kernel is not the installed one, 5.8.1-arch1-1 got outdated on Aug 16th.
What's the output of "ls /lib/modules"?

Khere · 2020-09-28 16:41:09

mount: /boot: unknown filesystem type 'vfat' 
boot.mount: Mount process exited, code=exited, status=32/n/a

There's the problem

Scimmia · 2020-09-28 17:07:33

Khere wrote:

mount: /boot: unknown filesystem type 'vfat' 
boot.mount: Mount process exited, code=exited, status=32/n/a

There's the problem

No, that's a symptom.

solskog · 2020-09-28 22:03:53

megthepeng wrote:

The only thing that changed was upgrading my kernel from 5.7 to 5.8.

OP's system booted with the 5.8 kernel but keeping a mismatched /lib/moudule/5.7 tree, hence failed to load vfat moudule.

Last edited by solskog (2020-09-28 22:07:21)

seth · 2020-09-28 22:11:24

That's the theory behind #10 - did he DM or irc you or are you jusat assuming this to be the case?

solskog · 2020-09-28 23:11:17

seth wrote:

did he DM or irc you or are you jusat assuming this to be the case?

No sorry, it was my assumption. Everything I known is on this thread. And I still don't understand this:

@megthepeng: ro root=UUID=f2e69ac4-645a-45ef-9fa0-9089541a3531 initrd=boot\initramfs-linux.img

How come the 'initrd=boot\initramfs-linux.img' referring to another directory in 'ESP/boot/' and not initrd=\initramfs-linux.img in 'ESP/' ?

Last edited by solskog (2020-09-29 00:30:55)

megthepeng · 2020-09-29 18:44:03

Thanks to everyone for the help, I managed to fix my system!

The issue is most likely that I had the kernel and the config files on the root partition (/dev/nvme0n1p5) in the /boot directory, while the EFI partition (/dev/nvme0n1p1) was also trying to mount itself in /boot.
I mounted only the root partition, moved everything from /boot to an external drive, moved initramfs-linux.img, initramfs-linux-fallback.img, vmlinuz-linux and refind_linux.conf back on /boot, then finally made a /boot/efi dir and mounted the EFI partition on /boot/efi (edited the fstab to reflect changes).
I also uninstalled rEFInd, reinstalled it with refind-install and did a full system upgrade.
Then I also changed the ro in my refind-linux.conf to rw, just to be sure. The system booted just fine afterwards.

Arch Linux

#1 2020-09-27 07:36:09

[SOLVED] systemd drops me into emergency shell after upgrade

#2 2020-09-27 07:51:00

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#3 2020-09-27 08:19:01

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#4 2020-09-27 08:56:29

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#5 2020-09-27 10:03:24

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#6 2020-09-27 11:26:23

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#7 2020-09-27 13:00:51

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#8 2020-09-27 14:26:55

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#9 2020-09-28 11:37:42

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#10 2020-09-28 16:23:29

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#11 2020-09-28 16:41:09

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#12 2020-09-28 17:07:33

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#13 2020-09-28 22:03:53

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#14 2020-09-28 22:11:24

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#15 2020-09-28 23:11:17

Re: [SOLVED] systemd drops me into emergency shell after upgrade

#16 2020-09-29 18:44:03

Re: [SOLVED] systemd drops me into emergency shell after upgrade

Board footer