Issue with booting recent Linux Kernel

Kartious · 2014-08-10 17:16:39

Ever since after the kernel 3.15.5-1 any recent versions after updating causes my system to fail to boot and dropping into an emergency console.

Kernel Version

archKartious 
>  ~ uname -a
Linux nickarch 3.15.5-1-ARCH #1 SMP PREEMPT Thu Jul 10 07:08:50 CEST 2014 x86_64 GNU/Linux

After looking at the Journal it seems that the system does not recognise the filesystem type "vfat" which is my ESP

Journalctl -b -1

Aug 10 14:50:16 nickarch systemd-fsck[232]: fsck.fat 3.0.26 (2014-03-07)
Aug 10 14:50:16 nickarch systemd-fsck[232]: /dev/sdc1: 345 files, 8979/130812 clusters
Aug 10 14:50:16 nickarch mount[264]: mount: unknown filesystem type 'vfat'
Aug 10 14:50:16 nickarch systemd[1]: boot-efi.mount mount process exited, code=exited status=32
Aug 10 14:50:16 nickarch systemd[1]: Failed to mount /boot/efi.
Aug 10 14:50:16 nickarch systemd[1]: Dependency failed for Local File Systems.
Aug 10 14:50:16 nickarch systemd[1]: Unit boot-efi.mount entered failed state.
Aug 10 14:50:16 nickarch systemd-fsck[228]: /dev/sda1: clean, 540132/30531584 files, 58237710/122096390 blocks
Aug 10 14:50:16 nickarch systemd-journal[172]: Permanent journal is using 640.0M (max allowed 1.0G, trying to leave 1.5G free of 5.1G available → c
Aug 10 14:50:16 nickarch systemd-journal[172]: Time spent on flushing to /var is 102.763ms for 927 entries.
Aug 10 14:50:16 nickarch kernel: usb 2-1: ep 0x83 - rounding interval to 64 microframes, ep desc says 80 microframes
Aug 10 14:50:16 nickarch kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
Aug 10 14:50:16 nickarch kernel: usb 4-1: new high-speed USB device number 2 using ehci-pci
Aug 10 14:50:16 nickarch systemd[1]: mnt-wa.mount mount process exited, code=exited status=32
Aug 10 14:50:16 nickarch systemd[1]: Failed to mount /mnt/wa.
Aug 10 14:50:16 nickarch systemd[1]: Unit mnt-wa.mount entered failed state.

Picture Kernel Configuration
So I decided to build a custom kernel and see if the module is configured to load when compiled.. which it was, so I decided to add the msdostools along with the vfat module just incase and compiled it.... After all the waiting I tried to install the custom kernel which resulted in the same issue again. I have also tried installing the linux-lts package which also has the same issue. May I pick your brains for a few ideas? I really hope not to stay on 3.15.5 forever...

Last edited by Kartious (2014-08-13 19:45:13)

lolilolicon · 2014-08-10 17:36:32

I think it won't take a long time to bisect 3.15.5 and 3.15.6, even though you need to reboot to test every time.

Kartious · 2014-08-10 20:14:11

Bisect as in comparing the two before compiling ? I can do that when I get some time.

ooo · 2014-08-10 20:31:33

do you have systemd in your mkinitcpio.conf HOOKS? i.e. is this happening at initrd phase or userspace?
could be that you're just missing vfat from your initrd for some reason...

if I'm not mistaken, your system should be able to boot just fine without mounting the ESP, as long as boot loader works from it.
try adding 'nofail' in your ESP's options in /etc/fstab, and your system should continue booting even if mounting of the partition fails.
Then you can try to mount it manually and diagnose further.

Last edited by ooo (2014-08-10 20:34:48)

lolilolicon · 2014-08-10 20:36:39

Refer to the manpage git-bisect(1), it's fairly straight-forward.
If you already compile your own kernel with minimal modules (e.g. via localmodconfig) it would make the process smoother/quicker. It helps if you have a custom kernel PKGBUILD handy.
The good news is that between 3.15.5 and 3.15.6 there aren't many patches going into the kernel so it's a small pool to catch the fish.

mrunion · 2014-08-11 14:49:55

WARNING: I may be WAYYYY off here!

I used to get similar messages before I had my ESP partition mounted in such a way that Kernel updates automatically installed to it. When I first installed Arch on my new laptop that used EFI, I had to remember to MANUALLY copy to the ESP the new Kernel images, etc. after a Kernel update. If I didn't, I would get the "vfat" error. Could this be your issue? Are you sure that after updating the Kernel, the new image/etc. is actually the stuff you're booting from?

Kartious · 2014-08-11 17:02:39

ooo wrote:

do you have systemd in your mkinitcpio.conf HOOKS?

No I don't I just use the generic config file when I run mkinitcpio

HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

Also putting nofail did not work. I got dropped into emergency mode again.

Journalctl -b -1 (boot with 3.15.8)

Aug 11 16:31:32 nickarch systemd-fsck[225]: /dev/sdc3: clean, 5012/675952 files, 1219655/2699264 blocks
Aug 11 16:31:32 nickarch systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway.
Aug 11 16:31:32 nickarch kernel: EXT4-fs (sdc3): mounted filesystem with ordered data mode. Opts: data=ordered
Aug 11 16:31:32 nickarch systemd-fsck[228]: fsck.fat 3.0.26 (2014-03-07)
Aug 11 16:31:32 nickarch systemd-fsck[228]: /dev/sdc1: 345 files, 8977/130812 clusters
Aug 11 16:31:32 nickarch mount[249]: mount: unknown filesystem type 'vfat'
Aug 11 16:31:32 nickarch systemd[1]: boot-efi.mount mount process exited, code=exited status=32
Aug 11 16:31:32 nickarch systemd[1]: Failed to mount /boot/efi.
Aug 11 16:31:32 nickarch systemd[1]: Unit boot-efi.mount entered failed state.
Aug 11 16:31:32 nickarch systemd-fsck[230]: /dev/sda1: clean, 542267/30531584 files, 58308134/122096390 blocks
Aug 11 16:31:32 nickarch kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
Aug 11 16:31:32 nickarch kernel: usb 2-1: ep 0x83 - rounding interval to 64 microframes, ep desc says 80 microframes
Aug 11 16:31:32 nickarch mount[259]: mount: /mnt/wa: mount failed: Unknown error -1
Aug 11 16:31:32 nickarch systemd[1]: mnt-wa.mount mount process exited, code=exited status=32
Aug 11 16:31:32 nickarch systemd[1]: Failed to mount /mnt/wa.
Aug 11 16:31:32 nickarch systemd[1]: Dependency failed for Local File Systems.
Aug 11 16:31:32 nickarch systemd[1]: Unit mnt-wa.mount entered failed state.
Aug 11 16:31:32 nickarch systemd-journal[174]: Permanent journal is using 664.0M (max allowed 1.0G, trying to leave 1.5G free of 5.1G available → current limit 1.0G).
Aug 11 16:31:32 nickarch systemd-journal[174]: Time spent on flushing to /var is 85.022ms for 935 entries.
Aug 11 16:31:33 nickarch kernel: usb 4-1: new high-speed USB device number 2 using ehci-pci
Aug 11 16:31:33 nickarch systemd-fsck[262]: /dev/sdb1: clean, 5582/15269888 files, 42588594/61049390 blocks
Aug 11 16:31:33 nickarch alsactl[267]: /usr/bin/alsactl: load_state:1729: No soundcards found...
Aug 11 16:31:33 nickarch kernel: hub 4-1:1.0: USB hub found
Aug 11 16:31:33 nickarch kernel: hub 4-1:1.0: 8 ports detected
Aug 11 16:31:33 nickarch systemd[269]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
Aug 11 16:31:33 nickarch kernel: EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: data=ordered
Aug 11 16:31:33 nickarch kernel: usb 4-1.5: new full-speed USB device number 3 using ehci-pci
Aug 11 16:31:33 nickarch kernel: Switched to clocksource tsc
Aug 11 16:31:33 nickarch kernel: usb 4-1.6: new full-speed USB device number 4 using ehci-pci

lolilolicon wrote:

Refer to the manpage git-bisect(1), it's fairly straight-forward.
If you already compile your own kernel with minimal modules (e.g. via localmodconfig) it would make the process smoother/quicker. It helps if you have a custom kernel PKGBUILD handy.
The good news is that between 3.15.5 and 3.15.6 there aren't many patches going into the kernel so it's a small pool to catch the fish.

Okay thanks for the heads up. I will look into this when I find some time, I will need to get 3.15.6 again but I can get that from the Arch Rollback Machine or get a friend (who is also on arch x86_64) to send it to me.

mrunion wrote:

I used to get similar messages before I had my ESP partition mounted in such a way that Kernel updates automatically installed to it. When I first installed Arch on my new laptop that used EFI, I had to remember to MANUALLY copy to the ESP the new Kernel images, etc. after a Kernel update. If I didn't, I would get the "vfat" error. Could this be your issue? Are you sure that after updating the Kernel, the new image/etc. is actually the stuff you're booting from?

Really? So where would the would the kernel images be if they are not already in the ESP ?

lsblk for reference

NAME   FSTYPE  LABEL UUID                                 MOUNTPOINT
sda                                                       
└─sda1 ext4          869b8001-7f92-48cd-bc46-816bb2201989 /home
sdb                                                       
└─sdb1 ext4          e2d17153-a82a-45fa-ba98-fdef32f94914 /media
sdc                                                       
├─sdc1 vfat          D92E-5388                            /boot/efi
├─sdc2 ext4          949416a7-abf3-48f4-bcc2-da0db7e49fab /
└─sdc3 ext4          b5f0d70f-8ce3-4c84-b1a1-373002444fa2 /var
sr0                                                       
loop0  iso9660 WA    2005-02-10-10-59-00-00               /mnt/wa

anatolik · 2014-08-11 17:31:44

Check that you have vfat kernel module successfully loaded:

lsmod | grep fat

if not then load it.

Kartious · 2014-08-12 20:11:26

anatolik wrote:

Check that you have vfat kernel module successfully loaded:
lsmod | grep fat
if not then load it.

I have tried adding the fat module to mkinitcpio and then installed 3.15.8, this made no difference.

Though I think I have may found some progress into the issue. I commented out all my fat partitions in fstab and then rebooted. The system managed to get into the login screen but the kernel version said 3.15.5-2 still which is similar to what mrunion has said.

Kartious · 2014-08-13 17:08:19

I am glad to say that I can mark this solved thanks to mrunion's advice.

For some reason when installing the new kernel the vmlinux-linux image is not being copied over (which is why I am getting dropped into emergency mode and also explains why I was still on 3.15.5 after updating). So after updating to the most recent version I extracted the package and manually copied over the file, rebooted and then vola!

Marking as solved, thanks for all your help guys.

EDIT: I just found out after fixing this and doing an Syu that 3.16-2 is out.... doh

Last edited by Kartious (2014-08-13 17:15:05)

mrunion · 2014-08-13 17:27:02

Sorry I missed the question about where the Kernel would be.

It is my understanding that it's still in "/boot/...", but that is not the "boot" on the ESP. You can mounts your $ESP at boot, and set your "$ESP boot" as "/boot" (it will 'overlay' your "regular" boot) and this problem will be solved. I would have to be on my Linux box to give the details, but a forum search might find it. I can;t remember the user that came up with that method, but smarter people than me will come along with the details, I'm sure.

Kartious · 2014-08-13 19:43:45

mrunion wrote:

It is my understanding that it's still in "/boot/...", but that is not the "boot" on the ESP. You can mounts your $ESP at boot, and set your "$ESP boot" as "/boot" (it will 'overlay' your "regular" boot) and this problem will be solved. I would have to be on my Linux box to give the details, but a forum search might find it. I can;t remember the user that came up with that method, but smarter people than me will come along with the details, I'm sure.

If you could do that, that would be great.

Unfortunately I think I may of spoken too soon. After upgrading to 3.16 I get an error I have never seen before complaining about a missing module (which I have never touched) and the system not being able to detect my root device. The UUID is correct:

Picture

lsblk -f

sda                                                             
└─sda1 ext4                869b8001-7f92-48cd-bc46-816bb2201989 /home
sdb                                                             
└─sdb1 ext4                e2d17153-a82a-45fa-ba98-fdef32f94914 /media
sdc                                                             
├─sdc1 vfat                D92E-5388                            /boot/efi
├─sdc2 ext4                949416a7-abf3-48f4-bcc2-da0db7e49fab /
└─sdc3 ext4                b5f0d70f-8ce3-4c84-b1a1-373002444fa2 /var
sdd    iso9660 ARCH_201405 2014-05-01-05-20-54-00               
├─sdd1 iso9660 ARCH_201405 2014-05-01-05-20-54-00               
└─sdd2 vfat    ARCHISO_EFI 96ED-5DA9                            
sde                                                             
└─sde1 vfat    Kingston    0548-6CFD                            /mnt/hdd
sr0                                                             
loop0  iso9660 WA          2005-02-10-10-59-00-00               /mnt/wa

I then decided to downgrade back to 3.15.8 and got the exact same missing module error (using the method you described which worked the first time). So I am back to 3.15.5 again... removing "[solved]"

Last edited by Kartious (2014-08-13 19:44:35)

mrunion · 2014-08-13 23:49:02

I'm not sure about the error that's happening now. It looks like the drive can't be found, not that there is a problem elsewhere.

As for the changes to the fstab, here is what I have:

# /dev/sda6
# Old line
#UUID=7249-22CA      	/boot/efi 	vfat      	noatime	0 2
UUID=7249-22CA      	/mnt/efi 	vfat      	noatime	0 2

# Bind Arch EFI Kernel location to /boot
/mnt/efi/EFI/arch	/boot		none bind	0 0

/dev/sda6 (my $ESP) is mounted at a mount point (you must create it first (mkdir /mnt/efi). Then /boot is "bound" to the actual Kernel boot from the ESP. Afer this change when you access /boot, you are actually accessing the $ESP becasue /boot is bound to that location through the /mnt/efu mount point.

Last edited by mrunion (2014-08-13 23:50:12)

Kartious · 2014-08-14 07:43:06

mrunion wrote:

I'm not sure about the error that's happening now. It looks like the drive can't be found, not that there is a problem elsewhere.
As for the changes to the fstab, here is what I have:
# /dev/sda6
# Old line
#UUID=7249-22CA      	/boot/efi 	vfat      	noatime	0 2
UUID=7249-22CA      	/mnt/efi 	vfat      	noatime	0 2

# Bind Arch EFI Kernel location to /boot
/mnt/efi/EFI/arch	/boot		none bind	0 0
/dev/sda6 (my $ESP) is mounted at a mount point (you must create it first (mkdir /mnt/efi). Then /boot is "bound" to the actual Kernel boot from the ESP. Afer this change when you access /boot, you are actually accessing the $ESP becasue /boot is bound to that location through the /mnt/efu mount point.

After some research and following this thread I have managed to fix the missing module issue and now managed to install the kernel 3.15.8-1 successfully by just installing the package. But this is where now it gets weird, if I install 3.16 through pacman and reboot I will again get dropped into emergency mode... and If I manually copy over the 3.16 kernel after running mkinitcpio as before I get the missing module error.... this is just a mess really.

I will look into the fstab changes, thanks for sharing.

Kartious · 2014-08-15 19:10:40

Right well, after trying a similar configuration to yours it hasn't fixed the issue, I just get the emergency console shoved to my face again.

/etc/fstab

#
# /etc/fstab: static file system information
#
# <file system>                                 <dir>           <type>                  <options>          <dump> <pass>
#/dev/sdc1
#UUID=D92E-5388                                 /boot/efi       vfat            defaults,nofail                 0 2


UUID=D92E-5388                                  /mnt/efi        vfat            noatime                         0 2

# Bind Arch EFI Kernel location to /boot
/mnt/efi/EFI/arch                               /boot           none            bind                            0 0



#/dev/sdc2
UUID=949416a7-abf3-48f4-bcc2-da0db7e49fab       /               ext4            rw,relatime,data=ordered        0 1

#/dev/sdc3
UUID=b5f0d70f-8ce3-4c84-b1a1-373002444fa2       /var            ext4            rw,relatime,data=ordered        0 2

#/dev/sda2
UUID=869b8001-7f92-48cd-bc46-816bb2201989       /home           ext4            rw,relatime,data=ordered        0 2

#/dev/sdb1
UUID=e2d17153-a82a-45fa-ba98-fdef32f94914       /media          ext4            rw,relatime,data=ordered        0 2

#Tmp under RAM
tmpfs                                           /tmp            tmpfs           nosuid,nodev,size=4G            0 0

# Worms Loop
/home/nick/wa.iso                           /mnt/wa         iso9660         ro,auto                         0 0

I then decided to try and manually rebuild the kernel image with mkinitcpio -p linux which then threw up the "most intresting error".... This is trying to rebuild the 3.16 kernel.

So now I check the /lib/modules directory...

archKartious
>  ~ ls /lib/modules
3.15.8-1-ARCH  extramodules-3.15-ARCH

Something definately isnt right here.

Edit: I have now successfully updated to the most recent kernel version 3.16.1-1. I chroot into the machine environment using an up to date USB ISO. Then ran an system update to 3.16 from 3.15.8, after I looked in /usr/lib/modules which had many different kernel version modules in the folder. I removed the out of date versions and rebooted.... which worked.

Unsure if this work around will work for the future so I will keep this thread open for a couple of future updates, or I will create a new thread and link this in.

Last edited by Kartious (2014-08-16 09:57:37)

Pse · 2014-08-17 02:43:58

Kartious wrote:

I then decided to try and manually rebuild the kernel image with mkinitcpio -p linux which then threw up the "most intresting error".... This is trying to rebuild the 3.16 kernel.

This error was probably caused because you did not specify for which kernel you were trying to build the image for. mkinitcpio chooses the current kernel (i.e. the kernel that was booted) if you don't pass a kernel version as an argument (which you probably needed if you booted from a live-cd and then chrooted). For future reference, when using mkinitcpio from a live-CD, use it like this:

mkinitcpio -p linux -k 3.16.1-1-ARCH

Last edited by Pse (2014-08-17 02:45:00)

Arch Linux

#1 2014-08-10 17:16:39

Issue with booting recent Linux Kernel

#2 2014-08-10 17:36:32

Re: Issue with booting recent Linux Kernel

#3 2014-08-10 20:14:11

Re: Issue with booting recent Linux Kernel

#4 2014-08-10 20:31:33

Re: Issue with booting recent Linux Kernel

#5 2014-08-10 20:36:39

Re: Issue with booting recent Linux Kernel

#6 2014-08-11 14:49:55

Re: Issue with booting recent Linux Kernel

#7 2014-08-11 17:02:39

Re: Issue with booting recent Linux Kernel

#8 2014-08-11 17:31:44

Re: Issue with booting recent Linux Kernel

#9 2014-08-12 20:11:26

Re: Issue with booting recent Linux Kernel

#10 2014-08-13 17:08:19

Re: Issue with booting recent Linux Kernel

#11 2014-08-13 17:27:02

Re: Issue with booting recent Linux Kernel

#12 2014-08-13 19:43:45

Re: Issue with booting recent Linux Kernel

#13 2014-08-13 23:49:02

Re: Issue with booting recent Linux Kernel

#14 2014-08-14 07:43:06

Re: Issue with booting recent Linux Kernel

#15 2014-08-15 19:10:40

Re: Issue with booting recent Linux Kernel

#16 2014-08-17 02:43:58

Re: Issue with booting recent Linux Kernel

Board footer