Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

digumx · 2021-05-26 16:57:18

Hi all,

Today, when trying to boot into my system, I was greeted with a message saying that `/boot/initramfs-linux.img` was not found, and the system failed to boot. I booted from a live USB, and so that the initramfs file was indeed missing, and after recreating it from inside a chroot, the system booted fine.

After rebooting, I found that some files directories in a non-root ntfs partition (`/dev/nvme0n1p1`, mounted at `/mnt/data`) was also corrupted.

I want to try to understand why this happened, and weather this is a one-off thing that is unlikely to repeat in the near future, or if this points to some underlying chronic issue with my hardware, software or system configuration.

The combination of these issues seems to point towards some sort of file system error, and indeed when I check `journalctl`, I get the following:

May 26 11:54:49 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4): ext4_validate_block_bitmap:390: comm bluetooth>
May 26 11:54:49 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 271938 at logical>
May 26 11:54:49 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:54:48 acernitro-arch rtkit-daemon[844]: The canary thread is apparently starving. Taking action.
May 26 11:54:49 acernitro-arch NetworkManager[486]: <warn>  [1622010289.5956] sup-iface[2c901d04722f94aa,3,wlp9s0]: cal>
May 26 11:55:00 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4): ext4_validate_inode_bitmap:105: comm Cache2 I/>
May 26 11:55:00 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Filesystem failed CRC
May 26 11:55:01 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:01 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:06 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem

After this point in the logs, several more `EXT4-fs error`s occur, all of them relating to the `nvme0n1p4` partition, which is the root partition. Here is another excerpt:

May 26 11:55:15 acernitro-arch kernel: EXT4-fs error: 131 callbacks suppressed
May 26 11:55:15 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:20 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4): ext4_validate_block_bitmap:390: comm kworker/u>
May 26 11:55:20 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 404486 at logical>
May 26 11:55:20 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:55:29 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:29 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:29 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 402499 at logical>
May 26 11:55:29 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:55:29 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 402503 at logical>
May 26 11:55:29 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:55:29 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:29 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 444257 at logical>
May 26 11:55:29 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:55:29 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:33 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:33 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 444263 at logical>
May 26 11:55:33 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:55:33 acernitro-arch kernel: EXT4-fs (nvme0n1p4): Delayed block allocation failed for inode 402499 at logical>
May 26 11:55:33 acernitro-arch kernel: EXT4-fs (nvme0n1p4): This should not happen!! Data will be lost
May 26 11:55:33 acernitro-arch kernel: EXT4-fs error (device nvme0n1p4) in ext4_free_inode:358: Corrupt filesystem
May 26 11:55:35 acernitro-arch kernel: EXT4-fs: 4 callbacks suppressed

Apart from this, several random applications seem to produce crash dumps.

There was data loss in the `/dev/nvme0n1p1` partition as well, however, searching for errors related to the `nvme0n1p1` partition in journalctl does not produce anything.

The full `journalctl` logs are provided here: https://www.filehosting.org/file/detail … 9/jctl.log

The machine I am using is rather new, it's less than 2 years old. It's an Acer Nitro 7 AN-715 laptop. The partitions are on a 512GB nvme ssd. `hwinfo` gives the following for the disk:

49: PCI 00.0: 10600 Disk
  [Created at block.245]
  Unique ID: wLCS.XvvbEczegk2
  Parent ID: vTuk._LHPI2mc8D6
  SysFS ID: /class/block/nvme0n1
  SysFS BusID: nvme0
  SysFS Device Link: /devices/pci0000:00/0000:00:1b.0/0000:06:00.0/nvme/nvme0
  Hardware Class: disk
  Model: "Sandisk WD Blue SN500 / PC SN520 NVMe SSD"
  Vendor: pci 0x15b7 "Sandisk Corp"
  Device: pci 0x5003 "WD Blue SN500 / PC SN520 NVMe SSD"
  SubVendor: pci 0x15b7 "Sandisk Corp"
  SubDevice: pci 0x5003
  Driver: "nvme"
  Driver Modules: "nvme"
  Device File: /dev/nvme0n1
  Device Files: /dev/nvme0n1, /dev/disk/by-id/nvme-eui.1925814481060001001b444a4458cc98, /dev/disk/by-path/pci-0000:06:00.0-nvme-1, /dev/disk/by-id/nvme-WDC_PC_SN520_SDAPNUW-512G-1014_192581448106
  Device Number: block 259:0
  Drive status: no medium
  Config Status: cfg=new, avail=yes, need=no, active=unknown
  Attached to: #38 (Non-Volatile memory controller)

Interestingly, when I try to run `hdparm -I /dev/nvme0n1` I get nothing:

[10:04:02] ~ $ sudo hdparm -I /dev/nvme0n1

/dev/nvme0n1:

Here is the output of `lsblk`:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.8T  0 disk
`-sda1        8:1    0   1.8T  0 part /mnt/media
nvme0n1     259:0    0 476.9G  0 disk
|-nvme0n1p1 259:1    0 260.9G  0 part /mnt/data
|-nvme0n1p2 259:2    0 120.9G  0 part /mnt/extdata
|-nvme0n1p3 259:3    0    16G  0 part [SWAP]
`-nvme0n1p4 259:4    0  78.9G  0 part /
nvme1n1     259:5    0 476.9G  0 disk
|-nvme1n1p1 259:6    0   100M  0 part /boot/efi
|-nvme1n1p2 259:7    0    16M  0 part
|-nvme1n1p3 259:8    0  99.5G  0 part
|-nvme1n1p4 259:9    0   520M  0 part
|-nvme1n1p5 259:10   0 343.8G  0 part
|-nvme1n1p6 259:11   0     1G  0 part
`-nvme1n1p7 259:12   0    32G  0 part [SWAP]

And, here is my `fstab`:

# /dev/nvme0n1p4 LABEL=ArchRoot
UUID=7f589701-6dc5-4b98-9bba-aff8f6e8e6c5	/         	ext4      	rw,relatime	0 1

# /dev/nvme1n1p1 LABEL=EFI
UUID=5C56-F8AE      	/boot/efi 	vfat      	rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

# /dev/nvme0n1p3 LABEL=swap
UUID=2b8805a8-de2b-4493-8401-b2531c096bd2	none      	swap      	defaults,pri=10  	0 0

# /dev/nvme0n1p3 LABEL=swap
UUID=6cef0cf5-920a-4141-a9ab-a1e25e38f976	none      	swap      	defaults,pri=10  	0 0

# /dev/nvme0n1p1 LABEL=SharedData
UUID=9CB09705B096E554   /mnt/data   ntfs-3g       uid=1000,gid=1001,umask=0003,fmask=0003,dmask=0003   0   0

# /dev/nvme0n1p2 LABEL=ExtendedData
UUID=194c4e98-7774-41d9-8e47-250b5e5df133   /mnt/extdata   ext4       defaults   0   0

I keep my system up to date.

Please let me know if there are any other logs that I can help you with, and let's hope we are able to understand something close to the cause of this failure.

Thank you in advance!

Slithery · 2021-05-26 17:00:12

Run a smart test on the drive and post the results.
https://wiki.archlinux.org/title/S.M.A.R.T.

digumx · 2021-05-26 20:54:47

Hi, thanks for the reply!

When I follow the wiki, `smartctl --info` does not give me any lines with "SMART support is: ..." for the device:

[02:17:12] ~ $ sudo smartctl --info --device=nvme /dev/nvme0n1
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.12.6-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       WDC PC SN520 SDAPNUW-512G-1014
Serial Number:                      192581448106
Firmware Version:                   20110000
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 4a4458cc98
Local Time is:                      Thu May 27 02:17:16 2021 IST

And, similarly, `smartctl -c` does not have the lines with the recommended polling times

[02:17:16] ~ $ sudo smartctl -c /dev/nvme0n1
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.12.6-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     86 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     3.00W       -        -    0  0  0  0        0       0
 1 +     2.60W       -        -    1  1  1  1        0       0
 2 +     1.70W       -        -    2  2  2  2        0       0
 3 -   0.0250W       -        -    3  3  3  3     5000    9000
 4 -   0.0025W       -        -    4  4  4  4     5000   44000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

However, smart still let's me start a test. I ran the long test via `smart -t long` and the output of `smart -x` after ~2 hours is as follows:

[02:19:06] ~ $ sudo smartctl -x /dev/nvme0n1
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.12.6-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       WDC PC SN520 SDAPNUW-512G-1014
Serial Number:                      192581448106
Firmware Version:                   20110000
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 4a4458cc98
Local Time is:                      Thu May 27 02:20:47 2021 IST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     86 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     3.00W       -        -    0  0  0  0        0       0
 1 +     2.60W       -        -    1  1  1  1        0       0
 2 +     1.70W       -        -    2  2  2  2        0       0
 3 -   0.0250W       -        -    3  3  3  3     5000    9000
 4 -   0.0025W       -        -    4  4  4  4     5000   44000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        40 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    4%
Data Units Read:                    22,507,642 [11.5 TB]
Data Units Written:                 26,658,671 [13.6 TB]
Host Read Commands:                 357,804,918
Host Write Commands:                971,102,432
Controller Busy Time:               837
Power Cycles:                       1,023
Power On Hours:                     1,937
Unsafe Shutdowns:                   129
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

To me it seems that smart is not reporting any errors... but smart also doesn't explicitly tell me that smart testing is available and enabled on the device. Does this mean that smart testing is being run, and there are no errors, or is smart testing not available, and so smart is unable to report an y potential issues?

Thanks again.

seth · 2021-05-27 06:09:24

/boot isn't on the root drive, right?
You're also not mounting disacard or noatime (latter recentely caused some initramfs fuckups for a user) - but what's your trimming strategy and do you shutdown "cleanly" or do you "have to" press the power button at some point?

Also, because of the second drive and the NTFS data partition: is there a parallel windows installation?
=> 3rd link in my signature.

digumx · 2021-05-27 13:30:28

boot isn't on the root drive, right?

Yes, /boot is on the root drive, my ESP is mounted at /boot/efi. This should not be causing any issues right?

You're also not mounting disacard or noatime (latter recentely caused some initramfs fuckups for a user) - but what's your trimming strategy and do you shutdown "cleanly" or do you "have to" press the power button at some point?

I was not aware of SSD Trim, and had not set up any kind of trim strategy, or ran fstrim ever. I didn't have the discard mount option set, or the fstrim.timer service enabled. I will now run fstrim on all the filesystems on the disk, and will enable the fstrim.timer. Reading about TRIM online, I think this was proabably a bad mistake, and may have caused some performance degradation. But do you think it could have caused the files to be deleted as well?

Also, because of the second drive and the NTFS data partition: is there a parallel windows installation?
=> 3rd link in my signature.

I should have mentioned this in my original post, but I am running a dual boot. However, I'm quite sure fast-boot is disabled in windows. Also, I have not booted into windows in the past few months, during which I have been using the NTFS drive regularly, so I think that this may not be the issue, right?

seth · 2021-05-27 13:49:52

The question is whether actually two different drives are affected by the data loss at the same time (what contradicts the idea of a HW failure)

Overly aggressive trimming is prone to cause data loss. Not trimming isn't aiding the drives life expectation, but not typically the cause of data loss either.
https://wiki.archlinux.org/title/Solid_state_drive#TRIM

Better check whether it's disabled, but I don't think you'd cause data loss when not actually running the two OS concurrently. Though better safe than sorry.

For clarification: the FS corruption is so far a singular incident?

seth wrote:

do you shutdown "cleanly" or do you "have to" press the power button at some point?

digumx · 2021-05-27 14:30:40

The question is whether actually two different drives are affected by the data loss at the same time (what contradicts the idea of a HW failure)

Yeah, that is a bit strange, although both the partitions are on the same physical disk.

Overly aggressive trimming is prone to cause data loss. Not trimming isn't aiding the drives life expectation, but not typically the cause of data loss either.
https://wiki.archlinux.org/title/Solid_state_drive#TRIM

Okay, so that is ruled out I guess. I will still keep the weekly trimming I think, or maybe tune it down to once a month. But that is a different issue.

Better check whether it's disabled, but I don't think you'd cause data loss when not actually running the two OS concurrently. Though better safe than sorry.

I will check this.

For clarification: the FS corruption is so far a singular incident?

Yeah, it is a singular incident. It has never happened on this disk before.

do you shutdown "cleanly" or do you "have to" press the power button at some point?

I actually rarely shutdown, but rather I suspend to disk. The suspend to disk process is clean, I have never had a situation where I have had to press the power button while shutting down, and I have never shut down by directly pressing the power button. I have configured my power button to suspend to disk as well, but I have never actually used the power button to suspend to disk.

Arch Linux

#1 2021-05-26 16:57:18

Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

#2 2021-05-26 17:00:12

Re: Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

#3 2021-05-26 20:54:47

Re: Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

#4 2021-05-27 06:09:24

Re: Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

#5 2021-05-27 13:30:28

Re: Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

#6 2021-05-27 13:49:52

Re: Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

#7 2021-05-27 14:30:40

Re: Random Error in Filesystem, Initramfs got Deleted, Other Files Corrupt

Board footer