[SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

maru · 2019-12-01 20:33:23

I'm not sure this is the right place to ask, but the problem is in the topic name.

I've had to downgrade my linux package to the previous one to keep using my system because after upgrading I've become unable to boot - there's some booting issue saying "Unable to mount /boot. unknown filesystem type 'vfat'".

Here I can see (https://www.archlinux.org/packages/core/x86_64/linux/) "Flagged out-of-date on 2019-11-25 Version 5.4.1.arch1-1 in testing" but don't really understand what it means - something tells me that I'm not the only one who faced issues with linux-5.3.13.1-1.

What's going on? Should I just wait for the next version of linux or try fixing it?

Edit: what I've tried so far:

Tried to reinstall grub out of chroot
Tried to reinstall linux package itself, and it sounds like it doesn't make any sense... but anyways.

Nothing's helped.

Edit: changed topic name so it can be helpful to others who will experience this kind of issue.

Last edited by maru (2019-12-02 02:09:09)

Trilby · 2019-12-01 21:04:27

maru wrote:

Unable to mount /boot. unknown filesystem type 'vfat'

Where does this message come from? What process emits it?

If the boot loader is unable to access the boot partition, the kernel is never loaded, and thus this is not a kernel issue. If the kernel boots and later fails to (re)mount the boot partition, this would not be a fatal error and you'd your system would run just fine (just updating the kernel later would be a problem).

maru wrote:

Here I can see (https://www.archlinux.org/packages/core/x86_64/linux/) "Flagged out-of-date on 2019-11-25

Yes, it was flagged in error. This is a common, but very annoying occurence that people click random buttons when they are unhappy for unrelated reasons.

maru wrote:

something tells me that I'm not the only one who faced issues with linux-5.3.13.1-1.

Could you ask that something to be more specific? I see no bug reports, and I don't recall similar threads being opened here.

I believe you are having trouble, but lets deal with diagnosing your problem not toss around vauge and false suspicions.

I'm not currently on an (u)efi system, but quite a large number of archers are. Given that efi partitions are all vfat, if this was a universal, or even common problem there'd have been a hell of a lot of noise about it already and the kernel would have been fixed or downgraded in the repos almost immediately. So lets assume this is not some global problem, and rather that there is a specific problem on your system.

Last edited by Trilby (2019-12-01 21:06:54)

maru · 2019-12-01 21:24:56

Trilby wrote:

maru wrote:
Unable to mount /boot. unknown filesystem type 'vfat'
Where does this message come from? What process emits it?

The message appears when my system boots.

Whatever it is, 5.3.13.1-1 refuses to work on my system, when the previous one does not. I don't know where to dig, I don't know why it fails to mount /boot. All I've done is just pacman -Syu, and then rebooted within a few days, and then faced this. I tried to reinstall linux package, I tried to reinstall grub, no more ideas.

I can provide any info you want.

Here's a screenshot of it: https://imgur.com/6LJ8mIE

Edit: I tried to upgrade one more time, and I got this: https://imgur.com/a/clvXhRM

This is all I can see about the issue. I've had to downgrade linux package again in order to edit this post.

Last edited by maru (2019-12-01 21:36:45)

Trilby · 2019-12-01 21:44:06

Ah, that's not at all what I expected from your description. The kernel boots fine. You are getting systemd messages of various things starting up. So the boot partition cannot be mounted in the running system which really should not be fatal - I don't know why systemd is going to emergency mode there.

Did you check the journal as instructed by the output at the end of that screenshot?

In emergency mode, can you `modprobe vfat`?

Last edited by Trilby (2019-12-01 21:47:36)

maru · 2019-12-01 21:55:45

Trilby wrote:

Ah, that's not at all what I expected from your description. The kernel boots fine. You are getting systemd messages of various things starting up. So the boot partition cannot be mounted in the running system which really should not be fatal - I don't know why systemd is going to emergency mode there.
Did you check the journal as instructed by the output at the end of that screenshot?
In emergency mode, can you `modprobe vfat`?

No, I can't modprobe vfat in emergency mode - module vfat not found in directory /lib/modules/5.3.12-arch1-1

Also, checked that journal - it's too large.

Last edited by maru (2019-12-01 21:57:14)

maru · 2019-12-01 22:16:10

And once I downgrade my linux kernel, I can modprobe vfat in emergency mode.

maru · 2019-12-01 22:21:50

in emergency mode I just downgraded my linux package and was able to modprobe vfat without any issues, then I just upgraded linux package again using the one from /var/cache/pacman/pkg, and rebooted successfully.

Thanks for the help.

Last edited by maru (2019-12-01 22:33:09)

Trilby · 2019-12-01 22:32:21

maru wrote:

... module vfat not found in directory /lib/modules/5.3.12-arch1-1

So you were not booting/running the installed kernel which is the result of not having the boot partition mounted when you first installed 5.3.13.

I'm glad all is well now (please remember to mark the thread as SOLVED by editing your first post). Keep in mind for the future, that when you have a problem, it is much more productive to explain your symptoms and assume there is an issue on your system rather than assuming there is a bug in a widely used bit of software (as well elaborated by esr).

V1del · 2019-12-01 22:35:21

Then that means you are booting the wrong kernel, this usually has two potential causes, you didn't have your /boot partition (as in the EFI partition) mounted or you did have the partition mounted but GRUB is configured to utilise the root partition instead of the EFI partition for booting. Find out what is what. I'd guess the first one to be more common, so go into a chroot with all partitions properly mounted and run a

pacman -Syu linux

in doubt post the outputs of

mount
cat /etc/fstab
lsblk -f

as well as your GRUB config.

Edit: Argh need to refresh more, still this might help figuring why this happened in the first place.

Last edited by V1del (2019-12-01 22:36:16)

ewaller · 2019-12-01 22:35:47

Maru,

Look at the version of the kernel that is running in post #5. It is 5.3.12. Note that the thread title says you are trying to run 5.3.13.

What is happening? When you installed 5.3.13, all of the kernel modules for 5.3.12 were removed and the ones for 5.3.13 were installed. But, for some reason, the kernel itself, that is supposed to be installed on the boot partition, was not. Your boot loader is still picking up 5.3.12. You never even tried to boot with the 5.3.13 kernel. So, when the 5.3.12 kernel gets to the point of trying to mount boot -- no problem -- it just looks in the /lib/modules/5.3.12-arch1-1 director to find that kernel module it need to read vfat files; wait, what? That directory does not exist? >Boom<.

What is odd, is that you do seem to be mounting a vfat directory (that you think is your boot partition) on /boot. I have a feeling that you are not booting from the partition you think you are.
Please confirm that this is a uEFI system, and not legacy BIOS.
Please post the output of:
lsblk
mount
ls -l /dev/disk/by-uuid/

If this is a uEFI system, the output of:
efibootmgr -v (requires efibootmgr package)

Edit: Dang, snaked by V1del yet again.

Last edited by ewaller (2019-12-01 22:36:32)

maru · 2019-12-01 22:40:43

Yes, it's indeed odd that when I modprobe'd vfat on 5.3.12 it was successfully done, and after upgrading again I was able to boot (most likely) 5.3.13.

I'm not sure if the issue is actually solved.

maru · 2019-12-01 22:46:02

V1del wrote:

Then that means you are booting the wrong kernel, this usually has two potential causes, you didn't have your /boot partition (as in the EFI partition) mounted or you did have the partition mounted but GRUB is configured to utilise the root partition instead of the EFI partition for booting. Find out what is what. I'd guess the first one to be more common, so go into a chroot with all partitions properly mounted and run a
pacman -Syu linux
in doubt post the outputs of
mount
cat /etc/fstab
lsblk -f
as well as your GRUB config.
Edit: Argh need to refresh more, still this might help figuring why this happened in the first place.

Since I was able to boot with 5.3.13 I guess I can do it without chrooting?

sudo pacman -Syu linux
[sudo] password for : 
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
warning: linux-5.3.13.1-1 is up to date -- reinstalling
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (1) linux-5.3.13.1-1

Total Installed Size:  72.49 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n]

cat /etc/fstab

cat /etc/fstab 
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sda1 UUID=830d4321-e872-45ca-86af-abb56830e153
LABEL=Arch          	/         	ext4      	discard,rw,relatime	0 1

# /dev/sda2 UUID=564B-6478
/dev/sda2           	/boot     	vfat      	discard,rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

# /dev/sda3 UUID=0322a2bc-e9a6-4bf0-ab3b-677db092ce2a
LABEL=Home          	/home     	ext4      	discard,rw,relatime	0 2

mount

mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=3746896k,nr_inodes=936724,mode=755)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on / type ext4 (rw,relatime,discard)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13508)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda3 on /home type ext4 (rw,relatime,discard)
/dev/sda2 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro,discard)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=758976k,mode=700,uid=1000,gid=1000)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

lsblk -f

NAME   FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda                                                                     
|-sda1 ext4   Arch  830d4321-e872-45ca-86af-abb56830e153   39.5G    22% /
|-sda2 vfat         564B-6478                             870.5M     9% /boot
`-sda3 ext4   Home  0322a2bc-e9a6-4bf0-ab3b-677db092ce2a     92G    28% /home
sr0

Please don't close this thread, I'll come here later.

Last edited by maru (2019-12-01 22:49:00)

maru · 2019-12-01 22:47:51

/dev/disk/by-uuid:

total 0
lrwxrwxrwx 1 root root 10 Dec  2 00:17 0322a2bc-e9a6-4bf0-ab3b-677db092ce2a -> ../../sda3
lrwxrwxrwx 1 root root 10 Dec  2 00:17 564B-6478 -> ../../sda2
lrwxrwxrwx 1 root root 10 Dec  2 00:17 830d4321-e872-45ca-86af-abb56830e153 -> ../../sda1

maru · 2019-12-01 22:51:34

uname -r

5.3.13-arch1-1

I guess the issue is solved? Not really sure, because all I've done is just modprobe vfat on 5.3.12 and then upgraded to 5.3.13, and rebooted successfully. I feel like my system now is using vfat module from 5.3.12 even though I'm booting 5.3.13. Makes no sense but still.

ewaller · 2019-12-01 22:55:17

what is the output of ls -l /lib/modules ?

maru · 2019-12-01 22:56:52

ewaller wrote:

what is the output of ls -l /lib/modules ?

ls -la /lib/modules

total 140
drwxr-xr-x   3 root root   4096 Dec  2 00:16 .
drwxr-xr-x 155 root root 131072 Dec  1 02:41 ..
drwxr-xr-x   3 root root   4096 Dec  2 00:16 5.3.13-arch1-1

ewaller · 2019-12-01 23:06:02

I think it is safe to say you are not using a 5.3.12 module. Only the ...13 modules exist.

Last edited by ewaller (2019-12-01 23:06:25)

Trilby · 2019-12-01 23:31:27

I suspect all is well now. Somehow, on at least one occasion in the past, your boot partition either failed to mount or was unmounted (either case is generally harmless and could very easily go unnoticed). And you were in this state when you first upgraded to 5.13.13 so the subsequent reboot failed as described in the first post.

maru · 2019-12-02 02:13:15

Marked as solved. Thanks.

Arch Linux

#1 2019-12-01 20:33:23

[SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#2 2019-12-01 21:04:27

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#3 2019-12-01 21:24:56

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#4 2019-12-01 21:44:06

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#5 2019-12-01 21:55:45

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#6 2019-12-01 22:16:10

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#7 2019-12-01 22:21:50

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#8 2019-12-01 22:32:21

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#9 2019-12-01 22:35:21

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#10 2019-12-01 22:35:47

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#11 2019-12-01 22:40:43

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#12 2019-12-01 22:46:02

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#13 2019-12-01 22:47:51

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#14 2019-12-01 22:51:34

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#15 2019-12-01 22:55:17

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#16 2019-12-01 22:56:52

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#17 2019-12-01 23:06:02

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#18 2019-12-01 23:31:27

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

#19 2019-12-02 02:13:15

Re: [SOLVED] upgraded linux kernel issue at boot (unknown filesystem vfat)

Board footer