[SOLVED] Kernel 5.15.77-1 and Kernel 6.0.7-1 vFat module issue

JoeAnto · 2022-11-05 19:25:18

Problem:

Systemd unable to mount \efi because the vfat kernel module is missing at boot

Steps taken:

nuked /boot and /efi and installed latest GRUB instead of reFind thinking that something might have gone wrong with it, since it's not as well supported as GRUB.
cleaned up /usr/lib/modules/ and reinstalled linux and linux-lts along with the corresponding linux headers for each kernel.
tried to roll the lts kernel back, didn't work, so I updated kernels to the latest version again

Current state:

The EFI partition is disabled in /etc/fstab, and the system boots with interesting quirks like the sound not working, but bluetooth audio is working as usual. Doing modprobe vfat loads the kernel module with no problems, and I am able to mount the EFI partition to /efi.

UPDATE

These are the various command outputs:

pacman -Q linux linux-lts

linux 6.0.7.arch1-1
linux-lts 5.15.77-1

uname -r

5.15.77-1-lts

lsblk -f

NAME   FSTYPE FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sdb
├─sdb1 vfat   FAT32 BOOT     C457-B9A6                             391.7M    23% /boot
├─sdb2 ext4   1.0   Root     9c0483c1-a7d4-4bfa-855b-9f5d2721c08f   33.2G    61% /
├─sdb3 ext4   1.0   Home     da6cfb0a-83dc-41bd-bd98-515e64bef670   41.2G    58% /home
└─sdb4 swap   1     Swap     43ed733e-d470-4bc3-ab92-98dd0dc4aeaf                [SWAP]

and GRUB is installed correctly using the command

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=ArchLinux

After checking out the note Here, as well as the known bug Here, I created the relevant file in /etc/modules-load.d, and changed the EFI partition label to BOOT instead of EFI. I also mapped the FAT32 partition to the /boot directory instead of just the efi directory.

Now, the same issue persists, but the vfat module is available to me when I comment out the EFI entry from /etc/fstab and try to mount the partition after the system boots. I'm out of ideas at this point.

UPDATE 2

After some probing, some kernel modules appear to be unavailable by default, requiring me to manually modprobe the module and then restart the respective daemon.
Example: the tun module required by tailscale. I also suspect that an audio-related kernel module is also having issues.

Feel free to request any missing data. I included stuff that I felt are relevant to the issue.

UPDATE 3

For anyone looking for a solution

Weirdly enough, the system started behaving as normal after updating the dkms package a couple of minutes ago and regenerating initrd with mkinitcpio. I'll mark the topic as [SOLVED] despite not knowing what actually happened.

The notable packages that got updated prior to the system working properly:
lib32-openssl-1.1
openssl-1.1
dkms

Last edited by JoeAnto (2022-11-06 15:27:33)

Scimmia · 2022-11-05 19:37:13

This usually means your /boot partition wasn't mounted when you updated. Check uname -r, I bet it doesn't match what's installed with pacman.

astralc · 2022-11-05 19:37:43

do you have different /boot and /efi (and /) partitions?
Sound more like you may have some other issue - what is output of (3 commands):

pacman -Q linux linux-lts
uname -r
lsblk -f

JoeAnto · 2022-11-05 19:55:29

These are the various command outputs:

pacman -Q linux linux-lts

linux 6.0.7.arch1-1
linux-lts 5.15.77-1

uname -r

5.15.77-1-lts

lsblk -f

NAME   FSTYPE FSVER LABEL    UUID                                                           FSAVAIL FSUSE% MOUNTPOINTS
sdb
├─sdb1 vfat        FAT32    EFI       618A-2499                                                    510.8M     0%         /efi
├─sdb2 ext4       1.0         Root     9c0483c1-a7d4-4bfa-855b-9f5d2721c08f     192.9M    95%       /
├─sdb3 ext4       1.0         Home   da6cfb0a-83dc-41bd-bd98-515e64bef670    5.6G        90%       /home
└─sdb4 swap     1            Swap    43ed733e-d470-4bc3-ab92-98dd0dc4aeaf                                 [SWAP]

and GRUB is installed correctly, making use of the separate EFI partition using the command

grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id=ArchLinux

JoeAnto · 2022-11-05 19:57:53

astralc wrote:

do you have different /boot and /efi (and /) partitions?
Sound more like you may have some other issue - what is output of (3 commands):
pacman -Q linux linux-lts
uname -r
lsblk -f

Only root, efi, and home partitions, along with a swap partition. the efi partition is shown as mounted in the previous command output because I manually loaded the vfat module and mounted it after boot.

cfr · 2022-11-05 21:20:01

So when you say you 'nuked' /boot and /efi, what is /boot? Is that on your root partition?

Did you regenerate grub's config file?

JoeAnto · 2022-11-05 21:32:36

cfr wrote:

So when you say you 'nuked' /boot and /efi, what is /boot? Is that on your root partition?
Did you regenerate grub's config file?

Yes. I had a /boot in my root partition. I even tried changing the structure and mapping the whole /boot directory to the EFI partition and reinstalling kernels/headers and regenerating grub configs (that's what I'm currently running. check the updated main post.)
Same issues are present.

The main post has more useful information than my replies.

astralc · 2022-11-06 09:52:39

did you disabled any systemd service/triggers before it happen?
Also, to make sure it not from other stuff - do you use any libraries or important utils from aur or unofficial repo?

is automatically loading modules works? (but, as it written there, udev should handle it anyway)
https://wiki.archlinux.org/title/Kernel … le_loading

JoeAnto · 2022-11-06 12:35:38

astralc wrote:

did you disabled any systemd service/triggers before it happen?
Also, to make sure it not from other stuff - do you use any libraries or important utils from aur or unofficial repo?
is automatically loading modules works? (but, as it written there, udev should handle it anyway)
https://wiki.archlinux.org/title/Kernel … le_loading

Did not disable any Systemd services prior to the update. I usually only manage Systemd services that I manually install, like VPN/Mesh networking stuff.
As for other Utils, I don't have any unofficial repos enabled. And the most notable things from the AUR that are system critical are the Nvidia 470xx drivers because of my GTX760.

As for auto-loading, I don't know how to check if it's working properly or not.

dimich · 2022-11-06 13:03:51

JoeAnto wrote:

Systemd unable to mount \efi because the vfat kernel module is missing at boot

What is exact error message in journal? On what stage systemd tries to mount /efi, in initrd or in main rootfs? Have you tried to add vfat module to /etc/mkinitcpio.conf and regenerate inird?

JoeAnto · 2022-11-06 13:32:03

Weirdly enough, the system started behaving as normal after updating the dkms package a couple of minutes ago and regenerating initrd with mkinitcpio. I'll mark the topic as [SOLVED] despite not knowing what actually happened.

The notable packages that got updated prior to the system working properly:
lib32-openssl-1.1
openssl-1.1
dkms

Last edited by JoeAnto (2022-11-06 13:36:30)

Arch Linux

#1 2022-11-05 19:25:18