Arch Linux

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Kernel upgrade to 4.19.X breaks mounting process at startup

Hello,
A few days ago I upgraded to the 4.19 kernel (yay -Syu).
Everything was installed successfully. After each kernel upgrade I usually reboot, so I did this time.

When booting again it got stuck and dumped some error messages like:

Nov 14 20:07:27 xps systemd[1]: Failed to start Load Kernel Modules.
...
Nov 14 20:07:27 xps systemd[1]: Failed to mount /boot.

After receiving this error I had to reboot into the ISO.
What I did here was:
* (connect to internet and remount partitions)
* reinstall the kernel
* mkinitcpio - p linux
* grub-mkconfig -o /boot/grub/grub.cfg

After rebooting, everything worked again. Now, updating to the latest kernel hotfix again this issue happened.

...
Nov 17 09:11:00 xps systemd[1]: boot.mount: Directory /boot to mount over is not empty, mounting anyway.
Nov 17 09:11:00 xps kernel: audit: type=1130 audit(1542222660.930:10): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-8474\x2d6AC3 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 17 09:11:00 xps systemd[1]: Mounting /boot...
Nov 17 09:11:00 xps mount[308]: mount: /boot: unknown filesystem type 'vfat'.
Nov 17 09:11:00 xps systemd[1]: boot.mount: Mount process exited, code=exited status=32
Nov 17 09:11:00 xps systemd[1]: boot.mount: Failed with result 'exit-code'.
Nov 17 09:11:00 xps systemd[1]: Failed to mount /boot.
...

Actually I do not want to fix this all the time, it is pretty annoying for me.
I am a quite "new" arch user, using it since 3 or 4 months, so bare with me if I do not get something instantly.

Any help is appreciated!
Thank you guys!

c00ter

Member

From: Alaskan in Washington State

Registered: 2014-08-28

Posts: 421

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Using AUR helpers/Pacman wrappers to run system upgrades is not a good idea.  https://github.com/Jguer/yay/commit/3bdb534

Last edited by c00ter (2018-11-17 15:39:09)

---

UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn

loqs

Member

Registered: 2014-03-06

Posts: 18,931

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Is /boot a seperate partition?  If yes is it included in the fstab?  Is the system booting using EFI or BIOS?  If EFI where is the ESP mounted?

Last edited by loqs (2018-11-17 15:59:35)

Torxed

Member

Registered: 2013-01-10

Posts: 203

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

I got the same problem, but it was due to a kernel panic in the mits of the installation. So maybe this won't be useful to you.
But what happned to me was that the kernel moduels (more specifically the ones in charge of file systems) didn't get installed properly.

So what I did was boot a live-cd, did `archroot /mnt` and did `pacman -S linux --force`.
The force is never recommended, but in my case I had to many empty files to patch everything manually.

That should get you able to boot the system.
You'll most likely get a lot of other issues, but re-installing your packages one by one or reinstall everything already installed might fix it.

I can elaborate my answer a little more in a bit. But that's the basic just of what I had to do.

Last edited by Torxed (2018-11-17 16:26:10)

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Is /boot a seperate partition?  If yes is it included in the fstab?  Is the system booting using EFI or BIOS?  If EFI where is the ESP mounted?

Yes, boot is a separate partition.
See fstab here:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p2
UUID=2ad8916f-087b-4fe5-be11-e9f9862a7e84	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p1
UUID=8474-6AC3      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

The system is booting with EFI and it actually must be mounted to /boot or am I wrong here?

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Using AUR helpers/Pacman wrappers to run system upgrades is not a good idea.  https://github.com/Jguer/yay/commit/3bdb534

Thanks for pointing that out!
Do you really think that the issue could be related to this? That it really breaks the kernel modules although the kernel upgrade was "successful"?

Using the flag --nocombinedupgrade should then avoid the auto update right?

loqs

Member

Registered: 2014-03-06

Posts: 18,931

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

The system is booting with EFI and it actually must be mounted to /boot or am I wrong here?

The ESP needing to be /boot depends on the bootloader in use.  If you unmount /boot is there a /boot/vmlinuz-linux present on the / filesystem?
If so please run file on it and post the output.

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,727

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

It sounds to me like grub is using the kernel from your root partition instead of the one on the ESP. Check the config.

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

The system is booting with EFI and it actually must be mounted to /boot or am I wrong here?

The ESP needing to be /boot depends on the bootloader in use.  If you unmount /boot is there a /boot/vmlinuz-linux present on the / filesystem?
If so please run file on it and post the output.

If I unmount /boot I can see the following entries:

    4 drwxr-xr-x  4 root root     4096 Nov 17 09:53 .
    4 drwxr-xr-x 17 root root     4096 Nov 11 20:53 ..
   28 -rw-r--r--  1 root root    28160 Oct 19 08:21 amd-ucode.img
    4 drwxr-xr-x  2 root root     4096 Nov 11 20:57 efi
    4 drwxr-xr-x  6 root root     4096 Nov 17 09:54 grub
29340 -rw-r--r--  1 root root 30043863 Nov 17 09:54 initramfs-linux-fallback.img
 7984 -rw-r--r--  1 root root  8174966 Nov 17 09:54 initramfs-linux.img
 1708 -rw-r--r--  1 root root  1747456 Aug 25 10:35 intel-ucode.img
 5704 -rw-r--r--  1 root root  5838720 Nov 13 22:15 vmlinuz-linux

File output on the linuz file leads to:

/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 4.19.2-arch1-1-ARCH (builduser@heftig-16038) #1 SMP PREEMPT Tue Nov 13 21:16:19 UTC 2018, RO-rootFS, swap_dev 0x5, Normal VGA

And as pointed out, here my grub.cfg: https://pastebin.com/3Ym2g71M

Just to note: I think @Scimmia is right in some points. When doing the reboot after the upgrade, it always told me that the previous kernel will be loaded, not the one that I have installed right now.
Any more inputs I can deliver you for helping me out?

loqs

Member

Registered: 2014-03-06

Posts: 18,931

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Regenerate /boot/grub/grub.cfg and compare the newly generated grub.cfg to the current one.

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Regenerate /boot/grub/grub.cfg and compare the newly generated grub.cfg to the current one.

Backup'ed the old one and generated a new one with grub-mkconfig -o /boot/grub/grub.cfg

Diffed the two files and they are identical. What else could it be?

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

It sounds to me like grub is using the kernel from your root partition instead of the one on the ESP. Check the config.

What config exactly? If you see my latest reply, you can see that I regenerated the grub.cfg and diffed it to the old one and the files are identical.
Any other things I can post to help me out?

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Any other ideas on how I could fix this issue? If I miss any other information about my system, please let me know and I'll post it asap.
I appreciate any help for this issue!

Thank you guys!

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,727

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Actually READ the grub config, see what it's doing.

It looks like you probably have 2 grub configs, one on the /boot partition and one on the root partition.

Last edited by Scimmia (2018-11-18 17:34:51)

lukasbash

Member

Registered: 2018-11-17

Posts: 10

Re: Kernel upgrade to 4.19.X breaks mounting process at startup

Actually READ the grub config, see what it's doing.

It looks like you probably have 2 grub configs, one on the /boot partition and one on the root partition.

Okay, I read the grub config file. For me it seems fine. Maybe I do not get it?
It links to initramfs and vmlinuz on /boot. Also when I "repaired" everything after the kernel upgrade manually it works flawlessly.
Also when my EFI partition is mounted to /boot, then I do not even see any other folder than the EFI one with the *.efi file. When unmounted I see exactly one grub config. Also searched the whole file system for another grub.cfg file but without success.

@Scimmia Can you probably tell me where this second config file could reside?

Thank you!

Last edited by lukasbash (2018-11-19 18:04:42)