[Solved] Arch Update - Boot Issue

Zeox101 · 2017-07-11 12:48:20

Edit: sorry I forgot to mention I have an UEFI system and use systemd boot. My bad

I'm having an issue where following an update (pacman -Syu) I encounter an error when rebooting the system. In the bootscreen I always receive some error involving /boot. Specifically the error originally said that it is an unknown filesystem type 'vfat'.

After googling around on this issue I "seemed" to have solved it by adding vfat to the modules in mkinitcpio.conf
Specifically I did "nano /etc/mkinitcpio.conf" to add in MODULES="vfat", this is what the MODULES section of mkinitcpio.conf looks like now

 # vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES="piix ide_disk reiserfs"
MODULES="vfat"

I couldn't test whether or not it worked right away because, I had just updated. So now Its been a month since then and I did pacman -Syu again today, and received a slightly different boot error. It no longer says unknown filesystem type vfat but now just says that it failed to mount /boot. This is the error as its shown in journactl :

Jul 11 08:07:27 VermillionCastle systemd[1]: Found device Samsung_SSD_850_EVO_500GB boot.
Jul 11 08:07:27 VermillionCastle systemd[1]: boot.mount: Directory /boot to mount over is not empty, mounting anyway.
Jul 11 08:07:27 VermillionCastle systemd[1]: Mounting /boot...
Jul 11 08:07:27 VermillionCastle systemd[1]: boot.mount: Mount process exited, code=exited status=32
Jul 11 08:07:27 VermillionCastle systemd[1]: Failed to mount /boot.
Jul 11 08:07:27 VermillionCastle systemd[1]: boot.mount: Unit entered failed state.
Jul 11 08:08:17 VermillionCastle systemd[1]: Starting Reboot...

Now the way I worked around the vfat issue before I added vfat to modules, was that I would boot the Live Environment USB and arch-chroot into my installation and pacman -S linux. Then I would be able to boot fine. This also worked now, I'm able to boot into Linux after doing pacman -S linux in the chroot environment.

I researched this issue back when it was a vfat error, and all the threads I had found on the issue that the unknown vfat filetype issue is resolved when rebooting the system, but now that its just failing to mount /boot I'm just stumped. However its stilled resolved by re-installing linux, so I suspect that I didn't really solve the original issue even though its not seemingly no longer a vfat issue.

Its a boot issue, so I looked at my fstab to see if something was wrong there : (sdb1=boot, sdb2=swap, sdb3=root, sdb4= home), but it seemed all fine to me. I'll stay post it anyway in case I'm overlooking something.

  GNU nano 2.8.5                                                                 File: /etc/fstab                                                                           

#
# /etc/fstab: static file system information
#
# <file system> <dir>   <type>  <options>       <dump>  <pass>
# /dev/sdb3
UUID=4e5b59a7-e854-4511-a0dd-9ce4c0c751de       /               ext4            rw,relatime,data=ordered        0 1

# /dev/sdb1
UUID=386C-BEB8          /boot           vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro    0 2

# /dev/sdb4
UUID=40d037ed-9b51-477d-9013-164f8f16f4a6       /home           ext4            rw,relatime,data=ordered        0 2

# /dev/sdb2
UUID=c3b9e0d2-7d40-4831-ae86-f31fb121ef8c       none            swap            defaults.discard        0 0

Anyway I would like some help resolving this if possible, I don't entirely mind having to chroot into the system to pacman -S linux once a month when I update, but its something I could absolutely do without. Also sorry if this post ends up being messy, I tried to provide as much info as possible without it getting too cluttered.

Disclaimers: I normally use Vi. But I like nano when I just need to do something quick. I'm also regretting having swap, I never use end up having to use swap memory. When I was installing Linux, I got to the part of the guide that talked about swap space, I read about it and didn't entirely understand what it was for at first, but I figured I would rather have it and not need it, then need it and not have it.

Last edited by Zeox101 (2017-08-03 13:54:00)

tom.ty89 · 2017-07-11 13:06:42

Because your bootloader is still loading the old kernel. Probably because /boot wasn't mounted when you pacman -Syu, so the new kernel is installed to the /boot directory of the root filesystem but not the "real /boot". When you reboot, the bootloader loads the old kernel, which will no longer find its modules under /lib because of the change in version number. The only modules can be loaded (during early boot) are those included in the initramfs.

You should probably leave the mkinitcpio.conf the initramfs images alone, instead you do these:
1. boot with the fallback initramfs
2. append earlymodules=vfat to the linux line of the boot entry (e.g. press 'e' on the boot menu)
3. make sure the /boot partition is mounted properly
4. reinstall all kernel packages you have installed (e.g. pacman -S linux)
5. reboot as usual

Last edited by tom.ty89 (2017-07-11 13:09:41)

Zeox101 · 2017-07-11 13:19:03

I'm fairly certain that /boot was mounted when I did pacman -Syu. I was in an active session when I did it, so /boot should of been mounted since I was able to boot. For instance, I'm currently using Arch right now and from the output of "mount" I can see

 /dev/sdb1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-r

I do agree however that it does seem like my bootloader is loading the old kernel.

As for your five steps, I think I covered steps 1,3,4, and 5 by chrooting from the live environment doing pacman -S linux there. So I do have the ability to boot normally now. I'm trying to find a way however to prevent the error from happening at all, so that I don't need to pacman -S linux everytime I update. (Would you still recommend I edit my boot entry to include earlymodules=vfat ?)

So I suppose now I need to figure out why my bootloader is loading the old kernel (I'm 99% sure /boot was mounted when I updated for the reason I stated above), and then how to fix it.

Last edited by Zeox101 (2017-07-11 13:21:53)

tom.ty89 · 2017-07-11 13:43:32

Well there are other possible reasons of similar issue. For example, grub is (re)installed without /boot mounted but then you have a /boot partition and you (re)install kernel with it mounted.

The rationale is the same anyway. The best way to eliminate / figure out this kind of problem is to locate the unused set of (newer) kernel and initramfs when the problem shows up.

The five steps allow you to fix it without booting an Arch ISO and chroot or so. Step 1 guarantee that the vfat module of version of the loaded kernel would be available during early boot, step 2 instruct the early init script to modprobe it so that you can perform step 3 for step 4. Step 5 make the fix take effect.

Last edited by tom.ty89 (2017-07-11 13:44:57)

Zeox101 · 2017-07-11 13:50:01

I see. Well anyway thanks, really informative! I'll try updating again in a week or so and attempt to figure it out then

Last edited by Zeox101 (2017-07-11 13:50:34)

Zeox101 · 2017-07-27 22:04:08

Ok so I think I may have figured out what was going on.

I've been updating semi-frequently (about every 5 days since I last posted) to see if I could figure out anything new, but each update happened with no issues. I also made sure to check that /boot was mounted correctly before each update.

Today however when I updated, for what appears to be no reason /dev/sda2 was mounted at /boot instead of /dev/sdb1. /dev/sda2 is the ESP for my windows installation (I have no idea why there is even a kernel at /dev/sda2, but I'm sure I must of done something to cause that at some point. I've never heard of phantom kernels that get spirited away to other partitions)

So what I'm guessing causes my issue is that whenever there is a linux kernel update, it updates in /dev/sda2, and when I reboot and it mounts /dev/sdb1, it causes an issue due to /dev/sdb1 still having the older kernel.

This is what makes the most sense to me, but I'm still holding some doubt because I'm not sure what causes /dev/sda2 to be "randomly" mounted instead of sdb1 during some boots (This is the first time I've noticed it happening). I guess now I'll wait for another update to the linux kernel and see what happens.

Last edited by Zeox101 (2017-07-27 22:05:16)

Lone_Wolf · 2017-07-27 22:12:09

Post lslbk -f please so we can see the actual UUIDs used.

Zeox101 · 2017-07-27 22:19:01

Of course, I don't know how I overlooked that lol. I just looked at my fstab and saw /dev/sdb1 listed and got stumped.

NAME   FSTYPE LABEL            UUID                                 MOUNTPOINT
sda                                                                 
├─sda1 ntfs   Windows RE tools 66E49DE1E49DB431                     
├─sda2 vfat                    386C-BEB8                            /boot
├─sda3                                                              
└─sda4 ntfs   Windows          2A7AA0EF7AA0B8C9                     
sdb                                                                 
├─sdb1 vfat                    386C-BEB8                            
├─sdb2 swap                    c3b9e0d2-7d40-4831-ae86-f31fb121ef8c [SWAP]
├─sdb3 ext4                    4e5b59a7-e854-4511-a0dd-9ce4c0c751de /
├─sdb4 ext4                    40d037ed-9b51-477d-9013-164f8f16f4a6 /home
├─sdb5 ntfs   Samsung          9062A48462A470A0                     
└─sdb6 ext4                    e091a045-beb7-41ea-8b1d-7909dddd4e86 
sdc                                                                 
├─sdc1                                                              
└─sdc2 ntfs   Storage          D66089176089000B

So I can see that sdb1 and sda2 have the same UUID. Again, I'm not sure how/when I caused that happen. I must of cloned one partition at some point, since that's the only way that I know of to get two partitions to have the same UUID.

Edit: So is that the normal reaction that Arch does when two partition have the same UUID as the one listed in fstab, it randomly picks one at each boot?

So I'm guessing I can fix this by changing one of the UUIDs using the tune2fs command?
edit: or does tune2fs not work for vfat filetypes, that's the gist I'm getting after googling some stuff (And first line of the man page) about tune2fs

Last edited by Zeox101 (2017-07-27 22:26:03)

Lone_Wolf · 2017-07-27 22:38:44

I've searched a bit and found HOWTO change the UUID and or label of a FAT/vfat partition on Linux .

change-fat16-partition-uuid confirms the mlabel command is the right tool for this.

Note that i'm just the person that noted the unusual uuid and thought : let's check all uuid on the off chance those 2 partitions having same uuid.
Your issue is interesting, i hope others can shed light on what linux does when 2 partitions share same uuid .

Zeox101 · 2017-07-27 23:01:25

Thanks!

I'll change the UUID of /dev/sda2 and that should fix everything. I'll hold off on marking the thread as solved until the next kernel update just to make sure everything works out.

Edit: Changed the UUID of /dev/sda2 and updated today with no issues. So I'll mark this as solved now. Thanks for the help guys!

Last edited by Zeox101 (2017-08-03 13:53:43)

Arch Linux

#1 2017-07-11 12:48:20

[Solved] Arch Update - Boot Issue

#2 2017-07-11 13:06:42

Re: [Solved] Arch Update - Boot Issue

#3 2017-07-11 13:19:03

Re: [Solved] Arch Update - Boot Issue

#4 2017-07-11 13:43:32

Re: [Solved] Arch Update - Boot Issue

#5 2017-07-11 13:50:01

Re: [Solved] Arch Update - Boot Issue

#6 2017-07-27 22:04:08

Re: [Solved] Arch Update - Boot Issue

#7 2017-07-27 22:12:09

Re: [Solved] Arch Update - Boot Issue

#8 2017-07-27 22:19:01

Re: [Solved] Arch Update - Boot Issue

#9 2017-07-27 22:38:44

Re: [Solved] Arch Update - Boot Issue

#10 2017-07-27 23:01:25

Re: [Solved] Arch Update - Boot Issue

Board footer