You are not logged in.
Hey guys,
I've built a new PC and been using it with Arch exclusively for around 2 weeks now. Yesterday after doing a system update with pacman -Syu I received a bunch of file level errors ("bad message" and other things I can't remember).
Unfortunately the errors were not logged in the pacman log, but here is a sample output of pacman -Qk. In total there are 938 of these errors.
❯ pacman -Qk | egrep "^warn"
warning: akonadi-calendar-tools: /usr/share/doc/HTML/es/konsolekalendar/ (Bad message)
warning: akonadi-calendar-tools: /usr/share/doc/HTML/es/konsolekalendar/index.cache.bz2 (Bad message)
warning: akonadi-calendar-tools: /usr/share/doc/HTML/es/konsolekalendar/index.docbook (Bad message)
warning: akonadi-import-wizard: /usr/share/doc/HTML/es/importwizard/ (Bad message)
warning: akonadi-import-wizard: /usr/share/doc/HTML/es/importwizard/index.cache.bz2 (Bad message)
warning: akonadi-import-wizard: /usr/share/doc/HTML/es/importwizard/index.docbook (Bad message)
warning: akregator: /usr/share/doc/HTML/es/akregator/ (Bad message)
warning: akregator: /usr/share/doc/HTML/es/akregator/index.cache.bz2 (Bad message)
warning: akregator: /usr/share/doc/HTML/es/akregator/index.docbook (Bad message)
warning: ark: /usr/share/doc/HTML/es/ark/ (Bad message)
warning: ark: /usr/share/doc/HTML/es/ark/index.cache.bz2 (Bad message)
warning: ark: /usr/share/doc/HTML/es/ark/index.docbook (Bad message)
warning: artikulate: /usr/share/doc/HTML/es/artikulate/ (Bad message)
warning: artikulate: /usr/share/doc/HTML/es/artikulate/index.cache.bz2 (Bad message)
warning: artikulate: /usr/share/doc/HTML/es/artikulate/index.docbook (Bad message)
❯ pacman -Qk 2>&1 | egrep "^warni*." | wc -l
938All bad files are exclusively located in the /usr/share/doc/HTML/es directory.
❯ cd /usr/share/doc/HTML/es
❯ ls
ls: reading directory '.': Bad messagejournalctl output:
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm zsh: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm zsh: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): htree_dirblock_to_tree:1003: inode #524450: comm zsh: Directory block failed checksum
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm zsh: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm zsh: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm zsh: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm zsh: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm less: checksumming directory block 0
Jun 21 09:52:10 ccraft-01 kernel: EXT4-fs error (device dm-2): __ext4_find_entry:1545: inode #524450: comm less: checksumming directory block 0I suspect this issue stems from my RAID1 and LVM on top of LUKS setup, but I have no idea how to approach this kind of issue.
I booted the Arch Live ISO and ran fsck on my root partition (dm-2) and some error was fixed, unfortunately don't have the output but executing fsck again reported "clean" filesystem. But then from within the Arch Live environment I mounted the root partition and tried to access /usr/share/doc/HTML/es, which resulted again in "Bad message". Unmounted and ran fsck again and an error was again reported and apparently fixed. So everytime I try to access that one specific directory it immediately marks my ext4 root partition with errors. I also used "fsck -p -c" to scan for bad blocks but no errors were found.
Here are some more details about my setup:
lsblk:
❯ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 931,5G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part /boot
└─nvme0n1p2 259:3 0 931G 0 part
└─md0 9:0 0 930,9G 0 raid1
└─cryptlvm 253:0 0 930,9G 0 crypt
├─vg0-swap 253:1 0 8G 0 lvm [SWAP]
├─vg0-root 253:2 0 80G 0 lvm /
└─vg0-home 253:3 0 842,9G 0 lvm /home
nvme1n1 259:2 0 931,5G 0 disk
├─nvme1n1p1 259:4 0 512M 0 part
└─nvme1n1p2 259:5 0 931G 0 part
└─md0 9:0 0 930,9G 0 raid1
└─cryptlvm 253:0 0 930,9G 0 crypt
├─vg0-swap 253:1 0 8G 0 lvm [SWAP]
├─vg0-root 253:2 0 80G 0 lvm /
└─vg0-home 253:3 0 842,9G 0 lvm /homemdstat:
❯ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
976104448 blocks super 1.2 [2/2] [UU]
bitmap: 2/8 pages [8KB], 65536KB chunk
unused devices: <none>fdisk -l:
❯ sudo fdisk -l
Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 970 EVO 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0DC50582-AB06-43BE-BE8D-292A606CCF85
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1050623 1048576 512M EFI System
/dev/nvme0n1p2 1050624 1953523711 1952473088 931G Linux RAID
Disk /dev/nvme1n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 970 EVO 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AC087D42-AF44-44A5-81CC-9EBA6FDBAFB8
Device Start End Sectors Size Type
/dev/nvme1n1p1 2048 1050623 1048576 512M EFI System
/dev/nvme1n1p2 1050624 1953523711 1952473088 931G Linux RAID
Disk /dev/md0: 930,89 GiB, 999530954752 bytes, 1952208896 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/cryptlvm: 930,87 GiB, 999514177536 bytes, 1952176128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/vg0-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/vg0-root: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/vg0-home: 842,87 GiB, 905021751296 bytes, 1767620608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytessmartctl nvme0n1:
❯ sudo smartctl -a /dev/nvme0n1
[sudo] password for simon:
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.12.12-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 970 EVO 1TB
Serial Number: S5H9NS1NB11292P
Firmware Version: 2B2QEXE7
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1.000.204.886.016 [1,00 TB]
Unallocated NVM Capacity: 0
Controller ID: 4
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1.000.204.886.016 [1,00 TB]
Namespace 1 Utilization: 67.658.522.624 [67,6 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 002538 5b0143ebc2
Local Time is: Mon Jun 21 10:08:55 2021 CEST
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x03): S/H_per_NS Cmd_Eff_Lg
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 85 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 6.20W - - 0 0 0 0 0 0
1 + 4.30W - - 1 1 1 1 0 0
2 + 2.10W - - 2 2 2 2 0 0
3 - 0.0400W - - 3 3 3 3 210 1200
4 - 0.0050W - - 4 4 4 4 2000 8000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 55 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0%
Data Units Read: 2.246.373 [1,15 TB]
Data Units Written: 419.309 [214 GB]
Host Read Commands: 3.656.233
Host Write Commands: 4.592.789
Controller Busy Time: 50
Power Cycles: 22
Power On Hours: 48
Unsafe Shutdowns: 4
Media and Data Integrity Errors: 0
Error Information Log Entries: 11
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 55 Celsius
Temperature Sensor 2: 65 Celsius
Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Loggedsmartctl nvme1n1:
❯ sudo smartctl -a /dev/nvme1n1
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.12.12-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 970 EVO 1TB
Serial Number: S5H9NS1NB11677F
Firmware Version: 2B2QEXE7
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1.000.204.886.016 [1,00 TB]
Unallocated NVM Capacity: 0
Controller ID: 4
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1.000.204.886.016 [1,00 TB]
Namespace 1 Utilization: 999.532.072.960 [999 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 002538 5b0143ed43
Local Time is: Mon Jun 21 10:09:09 2021 CEST
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x03): S/H_per_NS Cmd_Eff_Lg
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 85 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 6.20W - - 0 0 0 0 0 0
1 + 4.30W - - 1 1 1 1 0 0
2 + 2.10W - - 2 2 2 2 0 0
3 - 0.0400W - - 3 3 3 3 210 1200
4 - 0.0050W - - 4 4 4 4 2000 8000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 49 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0%
Data Units Read: 17.616 [9,01 GB]
Data Units Written: 2.421.995 [1,24 TB]
Host Read Commands: 182.937
Host Write Commands: 6.863.190
Controller Busy Time: 48
Power Cycles: 22
Power On Hours: 48
Unsafe Shutdowns: 4
Media and Data Integrity Errors: 0
Error Information Log Entries: 11
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 49 Celsius
Temperature Sensor 2: 54 Celsius
Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged1. How can I fix the existing ext4 filesystem error present at /usr/share/doc/HTML/es ?
2. What can I do to prevent these kind of errors in the future? What could be even causing it? Is it the Samsung NVMe SSDs? The RAID1 setup in combination with LUKS and LVM maybe? Should I switch to another FS, like btrfs? (was thinking to use btrfs in the first place but since I had no prior experience I stuck with ext4)
I'm really lost at this point and my Google fu is completely failing me, so I'd really appreciate any hints! Thanks so much.
Offline
I was able to fix the existing errors, looks like something funky is going on with my RAID1 array. I followed the RAID scrubbing guide described here: https://wiki.archlinux.org/title/RAID#Scrubbing
> echo check > /sys/block/md0/md/sync_action
> cat /sys/block/md0/md/mismatch_cnt
8448
> echo repair > /sys/block/md0/md/sync_actionAfter the repair was done I ran another check and mismatch_cnt was at 0. I booted Arch Live ISO and executed another fsck and this time it actually asked to repair specific inodes (apparently the nodes were ok but there was a checksum mismatch)
After all this I can now access /usr/share/doc/HTML/es again without errors and pacman -Qk reports no more warnings.
But the big question now is why did this happen and what can I do to prevent this issue in the future? I suspect my RAID setup is causing issues, because I remember now that last week I also got an error message from pacman -Sy that the "extra database is corrupted" or something along those lines. This already made me suspicious but everything seemed to work fine afterwards. But now with this ext4 issue appearing I'm certain there is a bigger underlying problem.
I'd be willing to re-install my system and try another setup. Only requirement is it has to be full disk encrypted with RAID1. I did also choose LVM because I never used it and I wanted to try the snapshot feature as well be able to easily create and resize volumes so that I could try out different filesystems in the future.
Appreciate any suggestions what to do, thanks!
Offline