Getting corrupt file system on /boot

Osiris · 2017-03-19 12:27:31

Hello,

I use an Intel SSD 600p NVM for / and /boot. / works just fine but on /boot I frequently get a corrupted file system. Bootloader is systemd-boot. Partition is GPT

Disk /dev/nvme0n1: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): DDD47D17-AD0E-4C63-ACB5-0F76971A6BB3
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   EF00  EFI System
   2         1050624       500118158   238.0 GiB   8300  Linux filesystem

I format the partition using mkfs.fat -F32 /dev/nvme0n1p1, mount it, install the stuff pacman -Syu linux linux-lts intel-ucode and bootctl install, umount, do an fsck using fsck.vfat -tv /dev/nvme0n1p1 and I get

root@marduk / # fsck.vfat -tv /dev/nvme0n1p1
fsck.fat 4.1 (2017-01-24)
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "mkfs.fat"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
      4096 bytes per cluster
        32 reserved sectors
First FAT starts at byte 16384 (sector 32)
         2 FATs, 32 bit entries
    524288 bytes per FAT (= 1024 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 1064960 (sector 2080)
    130812 data clusters (535805952 bytes)
32 sectors/track, 64 heads
      2048 hidden sectors
   1048576 sectors total
FATs differ but appear to be intact. Use which FAT ?
1) Use first FAT
2) Use second FAT
? 1
Cluster 1022 out of range (190032371 > 130813). Setting to EOF.
Cluster 1023 out of range (120693308 > 130813). Setting to EOF.
Cluster 1024 out of range (92896890 > 130813). Setting to EOF.
Cluster 1025 out of range (182676639 > 130813). Setting to EOF.
[...]
/vmlinuz-linux
  File size is 4962064 bytes, cluster chain length is 4186112 bytes.
  Truncating file to 4186112 bytes.
Checking for bad clusters.
Checking for unused clusters.
Reclaimed 190 unused clusters (778240 bytes).
Checking free cluster summary.
Free cluster summary wrong (114392 vs. really 114582)
1) Correct
2) Don't correct
? 1
Perform changes ? (y/n) y
/dev/nvme0n1p1: 17 files, 16230/130812 clusters

Sometimes the filesystem appears to be fine after, sometimes it's full of corrupt filenames.

Is it some hardware trouble like defect SSD? Or linux-SSD incompatibility? I'm completely puzzled and thankful for any ideas!

Kernel is Linux marduk 4.9.14-1-lts, up-to-date Arch. The root partition shows no errors in an forced fsck.ext4. I found nothing helpful in the journal.

Thanks!

Last edited by Osiris (2017-03-19 12:32:04)

Head_on_a_Stick · 2017-03-19 12:38:45

Have you tried re-creating a fresh filesystem on the /boot partition?

FAT is rather fragile at the best of times.

Osiris · 2017-03-19 12:40:30

Yes, I did mkfs.fat -F32 /dev/nvme0n1p1 a couple of times. Even tried to dd if=/dev/zero of=/dev/nvm0n1p1 to remove any traces of whatever.

xdee · 2017-03-20 03:28:02

I had the same problem with an Intel 600p SSD. The EFI partition would get corrupted every time pacman was run, however files copied to it using a simple 'cp' would work fine.

My solution at the time (~2 weeks ago) was to umount the boot partition, format it, run pacman, move vmlinuz/img files away from /boot, mount EFI partition into /boot again, then copy the kernel/img files into it (also update the new UUID in /etc/fstab).

A few days ago I ran the Intel SSD update tool on my system and updated the firmware for the SSD, however I haven't tested if pacman still trashes the FAT32 partition.

Last edited by xdee (2017-03-20 03:30:29)

Osiris · 2017-03-20 09:11:10

xdee wrote:

I had the same problem with an Intel 600p SSD. The EFI partition would get corrupted every time pacman was run, however files copied to it using a simple 'cp' would work fine.
My solution at the time (~2 weeks ago) was to umount the boot partition, format it, run pacman, move vmlinuz/img files away from /boot, mount EFI partition into /boot again, then copy the kernel/img files into it (also update the new UUID in /etc/fstab).

I use the GPT mount generator from systemd, so no fstab present.

A few days ago I ran the Intel SSD update tool on my system and updated the firmware for the SSD, however I haven't tested if pacman still trashes the FAT32 partition.

I really would appreciate you to tell us about results, as soon as the next kernel update comes in. I will also look into updating the firmware. My current version, according to smartctl is PSF100C.

Best,
Florian

xdee · 2017-03-21 09:59:17

I updated the kernel to 4.10.4-1 (was on 4.10.1 before) just to test and all went fine, no corruption on the boot partition anymore. It seems that the latest SSD firmware solves the problem.

My current firmware is PSF109C, which according to this page is the latest.

GourdCaptain · 2017-03-21 22:27:19

As someone who has been unable to run the upgrade ISO (it kernel panics on boot, following their instructions on how to set it up -_- ), adding "sync" to the mount flags on the /boot VFAT partition is a workaround that has been stable for me for two months.

GourdCaptain · 2017-03-23 03:18:53

Okay, having talked with Intel tech support, there's a way to upgrade your SSD's firmware without booting into the firmware upgrade ISO if you have issues like me. This is partially their instructions with an addition of my workarounds to some issues I encountered doing them. They've taken those issues into account and say they're going to fix the tool in the future to be easier to use this way.

1) The first step is to boot Linux (either from the live environment or the regular hard drive installation, then download the Intel SSD Firmware Update Tool ISO file from this link: https://downloadcenter.intel.com/download/26491

2) After downloading the ISO file, open a terminal window inside the live environment. Become root using sudo su - and create a loopback mount directory by running mkdir /mnt/loop

3) Next, loopback-mount the ISO file:
mount -o ro,loop /home/mint/Downloads/issdfut_2.1.7.iso /mnt/loop

4) Now, we need to obtain a decompressed copy of the initrd (RAM disk image) on which Intel's tools are located:

lzcat /mnt/loop/initrd > /root/initrd-uncompressed

5) Once this is done, unmount the ISO file using: umount /mnt/loop

6) Now, we need to loopback mount the uncompressed initrd file:

mount -o loop,ro /root/initrd-compressed /mnt/loop

GOURDCAPTAIN EDIT: The update tool is a 32-bit executable using QT4. So in order to execute it on 64-bit Arch, you're going to need to install lib32-qt4 from the AUR and then due to a bug involving QT4 and running a program using it as root when your X Server is running as the user (as is done anymore with opensource drivers, I think NVIDIA does it as root with the proprietary drivers), install the xorg-xwrapper package from AUR to force it to run as root temporarily. Be sure to uninstall that after this is over, and restart the X Server to let it kick in before continuing. If you do it without this step (xorg-xwrapper) and it doesn't work, just kill the useless grey window it spawns - it's harmless

7) Finally, to run the actual update tool, run the following two commands in order:

cd /mnt/loop/usr/bin

./issdfut

8) After accepting the license agreement, you should be able to update the firmware.

GOURDCAPTAIN EDIT:: It also doesn't successfully shut down the system after finishing (the program just closes), you can do that manually.

Last edited by GourdCaptain (2017-03-23 03:20:03)

Osiris · 2017-03-23 17:03:09

Just did the firmware upgrade. Went smooth and the recent pacman upgrade did not produced any errors on the boot partition. Seems to work now. Thanks everybody!

EDIT: Just experienced the same behavior again with kernel 4.11.5. Seems to be fixed by another firmware update.

Last edited by Osiris (2017-06-19 19:02:24)

Arch Linux

#1 2017-03-19 12:27:31

Getting corrupt file system on /boot

#2 2017-03-19 12:38:45

Re: Getting corrupt file system on /boot

#3 2017-03-19 12:40:30

Re: Getting corrupt file system on /boot

#4 2017-03-20 03:28:02

Re: Getting corrupt file system on /boot

#5 2017-03-20 09:11:10

Re: Getting corrupt file system on /boot

#6 2017-03-21 09:59:17

Re: Getting corrupt file system on /boot

#7 2017-03-21 22:27:19

Re: Getting corrupt file system on /boot

#8 2017-03-23 03:18:53

Re: Getting corrupt file system on /boot

#9 2017-03-23 17:03:09

Re: Getting corrupt file system on /boot

Board footer