[SOLVED] can this ext4 FS be saved?

gcb · 2024-09-14 17:54:58

I have a laptop that probably have either bad ram or ssd... but both do pass stress tests and i don't have means to easily replace either now :shrug.
(The irony: we specifically got models with ECC support and ram slots, but there's no sodimm ddr5 ecc memory to be found)

I noticed that sometimes playing media from firefox will crash IO (since many many months of arch updates ago, which is why i think it's hardware). When the issue happens, everything "works" but it is as if I removed the ssd. There's no IO happening anywhere. I can only work with what's on memory. And obviously, nothing shows up on any logs.

Since it have full disk encryption, it can be both ram or ssd. But all fleet have the same ssd lot, and this is the only laptop with gskills. Exact model is: F5-4800S3434A16G gskill Ripjaws DDR5 SO-DIMM DDR5-4800

Normally, when that happens, i just reboot (fs is checked every boot) and I never reported anything bad.

Until a few days ago when the computer was left on for 48h, and it was probably in this state for most of the time. Now the root partition is completely screwed. I can luks open it, but then:

e2fsck 1.47.1 (20-May-2024)
ext2fs_open2: Bad magic number in super-block
e2fsck: Superblock invalid, trying backup blocks...
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
Clear journal? no

/dev/mapper/lvmg-arch2023 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 1310797 has zero dtime.  Fix? no

Inodes that were part of a corrupted orphan linked list found.  Fix? no

Inode 1310800 was part of the orphaned inode list.  IGNORED.
Inode 1340008 extent tree (at level 2) could be narrower.  Optimize? no

Inode 1445265 seems to contain garbage.  Clear? no

Inode 1445265 is in use, but has dtime set.  Fix? no

Inode 1445265 has imagic flag set.  Clear? no

Inode 1445265 has a extra size (30302) which is invalid
Fix? no

Inode 1445265 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445265 passes checks, but checksum does not match inode.  Fix? no

Inode 1445266 seems to contain garbage.  Clear? no

Inode 1445266 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445266 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445266 is in use, but has dtime set.  Fix? no

Inode 1445266 has a extra size (18223) which is invalid
Fix? no

Inode 1445266 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445266, i_size is 13953153468103901452, should be 0.  Fix? no

Inode 1445266, i_blocks is 69745507234360, should be 0.  Fix? no

Inode 1445266 passes checks, but checksum does not match inode.  Fix? no

Inode 1445267 seems to contain garbage.  Clear? no

Inode 1445267 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445267 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445267 is in use, but has dtime set.  Fix? no

Inode 1445267 has imagic flag set.  Clear? no

Inode 1445267 has a extra size (48900) which is invalid
Fix? no

Inode 1445267, i_size is 17896241143510314735, should be 0.  Fix? no

Inode 1445267, i_blocks is 44281709292943, should be 0.  Fix? no

Inode 1445267 passes checks, but checksum does not match inode.  Fix? no

Inode 1445268 seems to contain garbage.  Clear? no

Inode 1445268 seems to have inline data but extent flag is set.
Fix? no

Inode 1445268 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445268 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445268 is in use, but has dtime set.  Fix? no

Inode 1445268 has a extra size (26569) which is invalid
Fix? no

Inode 1445268 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445268 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445268 has an invalid root node.
Clear HTree index? no

Inode 1445268, i_size is 3638960123753951805, should be 0.  Fix? no

Inode 1445268, i_blocks is 250544652748502, should be 0.  Fix? no

Inode 1445268 passes checks, but checksum does not match inode.  Fix? no

Inode 1445269 seems to contain garbage.  Clear? no

Inode 1445269 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445269 is in use, but has dtime set.  Fix? no

Inode 1445269 has a extra size (64333) which is invalid
Fix? no

Inode 1445269 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445269 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445269 has an invalid root node.
Clear HTree index? no

Inode 1445269, i_size is 7696996017889674819, should be 0.  Fix? no

Inode 1445269, i_blocks is 94760663932439, should be 0.  Fix? no

Inode 1445269 passes checks, but checksum does not match inode.  Fix? no

Inode 1445270 seems to contain garbage.  Clear? no

Inode 1445270 seems to have inline data but extent flag is set.
Fix? no

Inode 1445270 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445270 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445270 is in use, but has dtime set.  Fix? no

Inode 1445270 has a extra size (14117) which is invalid
Fix? no

Inode 1445270 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445270 has an invalid root node.
Clear HTree index? no

Inode 1445270, i_size is 10734290719344546811, should be 0.  Fix? no

Inode 1445270, i_blocks is 22780051815535, should be 0.  Fix? no

Inode 1445270 passes checks, but checksum does not match inode.  Fix? no

Inode 1445271 seems to contain garbage.  Clear? no

Inode 1445271 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445271 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445271 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445271 is in use, but has dtime set.  Fix? no

Inode 1445271 has imagic flag set.  Clear? no

Inode 1445271 has a extra size (47012) which is invalid
Fix? no

Inode 1445271 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445271, i_size is 2042967931595350615, should be 0.  Fix? no

Inode 1445271, i_blocks is 46435201616518, should be 0.  Fix? no

Inode 1445271 passes checks, but checksum does not match inode.  Fix? no

Inode 1445272 seems to contain garbage.  Clear? no

Inode 1445272 is in use, but has dtime set.  Fix? no

Inode 1445272 has imagic flag set.  Clear? no

Inode 1445272 has a extra size (1104) which is invalid
Fix? no

Inode 1445272 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445272 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445272 has an invalid root node.
Clear HTree index? no

Inode 1445272, i_size is 758592830588130788, should be 0.  Fix? no

Inode 1445272, i_blocks is 96001720090043, should be 0.  Fix? no

Inode 1445272 passes checks, but checksum does not match inode.  Fix? no

Inode 1445273 seems to contain garbage.  Clear? no

Inode 1445273 is in use, but has dtime set.  Fix? no

Inode 1445273 has a extra size (8335) which is invalid
Fix? no

Inode 1445273 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445273, i_size is 13751455415639536449, should be 0.  Fix? no

Inode 1445273, i_blocks is 80812136811390, should be 0.  Fix? no

Inode 1445273 passes checks, but checksum does not match inode.  Fix? no

Inode 1445274 seems to contain garbage.  Clear? no

Inode 1445274 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445274 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445274 is in use, but has dtime set.  Fix? no

Inode 1445274 has imagic flag set.  Clear? no

Inode 1445274 has a extra size (50577) which is invalid
Fix? no

Inode 1445274 passes checks, but checksum does not match inode.  Fix? no

Inode 1445275 seems to contain garbage.  Clear? no

Inode 1445275 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445275 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445275 is in use, but has dtime set.  Fix? no

Inode 1445275 has a extra size (50635) which is invalid
Fix? no

Inode 1445275 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445275 has an invalid root node.
Clear HTree index? no

Inode 1445275, i_size is 17911352296934589865, should be 0.  Fix? no

Inode 1445275, i_blocks is 230130795500088, should be 0.  Fix? no

Inode 1445275 passes checks, but checksum does not match inode.  Fix? no

Inode 1445276 seems to contain garbage.  Clear? no

Inode 1445276 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445276 is in use, but has dtime set.  Fix? no

Inode 1445276 has imagic flag set.  Clear? no

Inode 1445276 has a extra size (50935) which is invalid
Fix? no

Inode 1445276 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445276 passes checks, but checksum does not match inode.  Fix? no

Inode 1445277 seems to contain garbage.  Clear? no

Inode 1445277 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445277 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445277 is in use, but has dtime set.  Fix? no

Inode 1445277 has imagic flag set.  Clear? no

Inode 1445277 has a extra size (19212) which is invalid
Fix? no

HTREE directory inode 1445277 has an invalid root node.
Clear HTree index? no

Inode 1445277, i_size is 9508001120151593297, should be 0.  Fix? no

Inode 1445277, i_blocks is 66537863091395, should be 0.  Fix? no

Inode 1445277 passes checks, but checksum does not match inode.  Fix? no

Inode 1445278 seems to contain garbage.  Clear? no

Inode 1445278 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445278 seems to have inline data but extent flag is set.
Fix? no

Inode 1445278 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445278 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445278 is in use, but has dtime set.  Fix? no

Inode 1445278 has imagic flag set.  Clear? no

Inode 1445278 has a extra size (13564) which is invalid
Fix? no

Inode 1445278 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445278, i_size is 4360572428163223825, should be 0.  Fix? no

Inode 1445278, i_blocks is 278135668532612, should be 0.  Fix? no

Inode 1445278 passes checks, but checksum does not match inode.  Fix? no

Inode 1445279 seems to contain garbage.  Clear? no

Inode 1445279 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445279 is in use, but has dtime set.  Fix? no

Inode 1445279 has a extra size (483) which is invalid
Fix? no

Inode 1445279, i_size is 1725069878534730551, should be 0.  Fix? no

Inode 1445279, i_blocks is 219818949606643, should be 0.  Fix? no

Inode 1445279 passes checks, but checksum does not match inode.  Fix? no

Inode 1445280 seems to contain garbage.  Clear? no

Inode 1445280 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445280 is in use, but has dtime set.  Fix? no

Inode 1445280 has a extra size (18231) which is invalid
Fix? no

Inode 1445280 passes checks, but checksum does not match inode.  Fix? no

Inode 1445345 seems to contain garbage.  Clear? no

Inode 1445345 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445345 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445345 seems to have inline data but extent flag is set.
Fix? no

Inode 1445345 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445345 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445345 is in use, but has dtime set.  Fix? no

Inode 1445345 has imagic flag set.  Clear? no

Inode 1445345 has a extra size (51699) which is invalid
Fix? no

Inode 1445345 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445345 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445345 has an invalid root node.
Clear HTree index? no

Inode 1445345, i_size is 13339184684343503610, should be 0.  Fix? no

Inode 1445345, i_blocks is 1014327926818, should be 0.  Fix? no

Inode 1445345 passes checks, but checksum does not match inode.  Fix? no

Inode 1445346 seems to contain garbage.  Clear? no

Inode 1445346 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445346 is in use, but has dtime set.  Fix? no

Inode 1445346 has a extra size (42161) which is invalid
Fix? no

Inode 1445346, i_size is 8397515578239917098, should be 0.  Fix? no

Inode 1445346, i_blocks is 11549210118187, should be 0.  Fix? no

Inode 1445346 passes checks, but checksum does not match inode.  Fix? no

Inode 1445347 seems to contain garbage.  Clear? no

Inode 1445347 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445347 is in use, but has dtime set.  Fix? no

Inode 1445347 has imagic flag set.  Clear? no

Inode 1445347 has a extra size (42486) which is invalid
Fix? no

Inode 1445347 has corrupt extent header.  Clear inode? no

Inode 1445347, i_size is 781529362698406547, should be 0.  Fix? no

Inode 1445347, i_blocks is 109748656842837, should be 0.  Fix? no

Inode 1445347 passes checks, but checksum does not match inode.  Fix? no

Inode 1445348 seems to contain garbage.  Clear? no

Inode 1445348 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445348 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445348 is in use, but has dtime set.  Fix? no

Inode 1445348 has imagic flag set.  Clear? no

Inode 1445348 has a extra size (64833) which is invalid
Fix? no

Inode 1445348 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445348, i_size is 6738573408816135261, should be 0.  Fix? no

Inode 1445348, i_blocks is 18788514601754, should be 0.  Fix? no

Inode 1445348 passes checks, but checksum does not match inode.  Fix? no

Inode 1445349 seems to contain garbage.  Clear? no

Inode 1445349 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445349 is in use, but has dtime set.  Fix? no

Inode 1445349 has imagic flag set.  Clear? no

Inode 1445349 has a extra size (49750) which is invalid
Fix? no

Inode 1445349, i_size is 7595691202941429918, should be 0.  Fix? no

Inode 1445349, i_blocks is 252422248138641, should be 0.  Fix? no

Inode 1445349 passes checks, but checksum does not match inode.  Fix? no

Inode 1445350 seems to contain garbage.  Clear? no

Inode 1445350 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445350 seems to have inline data but extent flag is set.
Fix? no

Inode 1445350 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445350 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445350 is in use, but has dtime set.  Fix? no

Inode 1445350 has a extra size (13466) which is invalid
Fix? no

Inode 1445350 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445350 has corrupt extent header.  Clear inode? no

Inode 1445350 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445350 has an invalid root node.
Clear HTree index? no

Inode 1445350, i_blocks is 248191149642008, should be 0.  Fix? no

Inode 1445350 passes checks, but checksum does not match inode.  Fix? no

Inode 1445351 seems to contain garbage.  Clear? no

Inode 1445351 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445351 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445351 is in use, but has dtime set.  Fix? no

Inode 1445351 has a extra size (54662) which is invalid
Fix? no

Inode 1445351 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445351 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445351 has an invalid root node.
Clear HTree index? no

Inode 1445351, i_size is 11102925901195676453, should be 0.  Fix? no

Inode 1445351, i_blocks is 74763515710332, should be 0.  Fix? no

Inode 1445351 passes checks, but checksum does not match inode.  Fix? no

Inode 1445352 seems to contain garbage.  Clear? no

Inode 1445352 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445352 is in use, but has dtime set.  Fix? no

Inode 1445352 has a extra size (61431) which is invalid
Fix? no

Inode 1445352 passes checks, but checksum does not match inode.  Fix? no

Inode 1445353 seems to contain garbage.  Clear? no

Inode 1445353 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445353 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445353 seems to have inline data but extent flag is set.
Fix? no

Inode 1445353 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445353 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445353 is in use, but has dtime set.  Fix? no

Inode 1445353 has a extra size (36657) which is invalid
Fix? no

Inode 1445353, i_size is 5055528838028824191, should be 0.  Fix? no

Inode 1445353, i_blocks is 69443120911908, should be 0.  Fix? no

Inode 1445353 passes checks, but checksum does not match inode.  Fix? no

Inode 1445354 seems to contain garbage.  Clear? no

Inode 1445354 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445354 is in use, but has dtime set.  Fix? no

Inode 1445354 has a extra size (15769) which is invalid
Fix? no

Inode 1445354, i_size is 7012604956692509563, should be 0.  Fix? no

Inode 1445354, i_blocks is 181798638568536, should be 0.  Fix? no

Inode 1445354 passes checks, but checksum does not match inode.  Fix? no

Inode 1445355 seems to contain garbage.  Clear? no

Inode 1445355 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445355 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445355 is in use, but has dtime set.  Fix? no

Inode 1445355 has a extra size (51774) which is invalid
Fix? no

Inode 1445355 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445355 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445355 has an invalid root node.
Clear HTree index? no

Inode 1445355, i_size is 4071343742061726721, should be 0.  Fix? no

Inode 1445355, i_blocks is 107502760521644, should be 0.  Fix? no

Inode 1445355 passes checks, but checksum does not match inode.  Fix? no

Inode 1445356 seems to contain garbage.  Clear? no

Inode 1445356 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445356 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445356 is in use, but has dtime set.  Fix? no

Inode 1445356 has a extra size (61839) which is invalid
Fix? no

Inode 1445356, i_size is 694948298930147150, should be 0.  Fix? no

Inode 1445356, i_blocks is 222386710825466, should be 0.  Fix? no

Inode 1445356 passes checks, but checksum does not match inode.  Fix? no

Inode 1445357 seems to contain garbage.  Clear? no

Inode 1445357 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445357 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445357 is in use, but has dtime set.  Fix? no

Inode 1445357 has a extra size (47528) which is invalid
Fix? no

Inode 1445357 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445357, i_size is 3123392006446478563, should be 0.  Fix? no

Inode 1445357, i_blocks is 100011362666075, should be 0.  Fix? no

Inode 1445357 passes checks, but checksum does not match inode.  Fix? no

Inode 1445358 seems to contain garbage.  Clear? no

Inode 1445358 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445358 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445358 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445358 is in use, but has dtime set.  Fix? no

Inode 1445358 has imagic flag set.  Clear? no

Inode 1445358 has a extra size (43413) which is invalid
Fix? no

Inode 1445358 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445358 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445358 has an invalid root node.
Clear HTree index? no

Inode 1445358, i_size is 12301790467498776724, should be 0.  Fix? no

Inode 1445358, i_blocks is 225542571225528, should be 0.  Fix? no

Inode 1445358 passes checks, but checksum does not match inode.  Fix? no

Inode 1445359 seems to contain garbage.  Clear? no

Inode 1445359 seems to have inline data but extent flag is set.
Fix? no

Inode 1445359 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445359 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445359 is in use, but has dtime set.  Fix? no

Inode 1445359 has imagic flag set.  Clear? no

Inode 1445359 has a extra size (30166) which is invalid
Fix? no

Inode 1445359 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445359, i_size is 22596597754399183, should be 0.  Fix? no

Inode 1445359, i_blocks is 197000954264752, should be 0.  Fix? no

Inode 1445359 passes checks, but checksum does not match inode.  Fix? no

Inode 1445360 seems to contain garbage.  Clear? no

Inode 1445360 is in use, but has dtime set.  Fix? no

Inode 1445360 has a extra size (2242) which is invalid
Fix? no

Inode 1445360 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445360 passes checks, but checksum does not match inode.  Fix? no

Inode 1445489 seems to contain garbage.  Clear? no

Inode 1445489 seems to have inline data but extent flag is set.
Fix? no

Inode 1445489 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445489 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445489 is in use, but has dtime set.  Fix? no

Inode 1445489 has imagic flag set.  Clear? no

Inode 1445489 has a extra size (57990) which is invalid
Fix? no

Inode 1445489 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445489 has an invalid root node.
Clear HTree index? no

Inode 1445489, i_size is 6597916636275285461, should be 0.  Fix? no

Inode 1445489, i_blocks is 36461163657807, should be 0.  Fix? no

Inode 1445489 passes checks, but checksum does not match inode.  Fix? no

Inode 1445490 seems to contain garbage.  Clear? no

Inode 1445490 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445490 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445490 is in use, but has dtime set.  Fix? no

Inode 1445490 has imagic flag set.  Clear? no

Inode 1445490 has a extra size (45163) which is invalid
Fix? no

Inode 1445490 passes checks, but checksum does not match inode.  Fix? no

Inode 1445491 seems to contain garbage.  Clear? no

Inode 1445491 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445491 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445491 is in use, but has dtime set.  Fix? no

Inode 1445491 has imagic flag set.  Clear? no

Inode 1445491 has a extra size (49147) which is invalid
Fix? no

Inode 1445491 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445491 has an invalid root node.
Clear HTree index? no

Inode 1445491, i_size is 12654146731372150140, should be 0.  Fix? no

Inode 1445491, i_blocks is 59977382056327, should be 0.  Fix? no

Inode 1445491 passes checks, but checksum does not match inode.  Fix? no

Inode 1445492 seems to contain garbage.  Clear? no

Inode 1445492 is in use, but has dtime set.  Fix? no

Inode 1445492 has a extra size (65389) which is invalid
Fix? no

Inode 1445492 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445492 passes checks, but checksum does not match inode.  Fix? no

Inode 1445493 seems to contain garbage.  Clear? no

Inode 1445493 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445493 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445493 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445493 is in use, but has dtime set.  Fix? no

Inode 1445493 has imagic flag set.  Clear? no

Inode 1445493 has a extra size (38288) which is invalid
Fix? no

Inode 1445493 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445493 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445493 has an invalid root node.
Clear HTree index? no

Inode 1445493, i_size is 5800232031778633362, should be 0.  Fix? no

Inode 1445493, i_blocks is 85178304644783, should be 0.  Fix? no

Inode 1445493 passes checks, but checksum does not match inode.  Fix? no

Inode 1445494 seems to contain garbage.  Clear? no

Inode 1445494 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445494 is in use, but has dtime set.  Fix? no

Inode 1445494 has imagic flag set.  Clear? no

Inode 1445494 has a extra size (50012) which is invalid
Fix? no

Inode 1445494 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445494 has an invalid root node.
Clear HTree index? no

Inode 1445494, i_size is 5427687654610309304, should be 0.  Fix? no

Inode 1445494, i_blocks is 261976775534935, should be 0.  Fix? no

Inode 1445494 passes checks, but checksum does not match inode.  Fix? no

Inode 1445495 seems to contain garbage.  Clear? no

Inode 1445495 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445495 is in use, but has dtime set.  Fix? no

Inode 1445495 has a extra size (37636) which is invalid
Fix? no

Inode 1445495 has encrypt flag but no encryption extended attribute.
Clear flag? no

Special (device/socket/fifo) inode 1445495 has non-zero size.  Fix? no

Inode 1445495 passes checks, but checksum does not match inode.  Fix? no

Inode 1445496 seems to contain garbage.  Clear? no

Inode 1445496 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445496 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445496 is in use, but has dtime set.  Fix? no

Inode 1445496 has a extra size (35261) which is invalid
Fix? no

Inode 1445496 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445496 has corrupt extent header.  Clear inode? no

Inode 1445496, i_blocks is 97287691777789, should be 0.  Fix? no

Inode 1445496 passes checks, but checksum does not match inode.  Fix? no

Inode 1445497 seems to contain garbage.  Clear? no

Inode 1445497 is in use, but has dtime set.  Fix? no

Inode 1445497 has imagic flag set.  Clear? no

Inode 1445497 has a extra size (3473) which is invalid
Fix? no

Inode 1445497 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445497 has an invalid root node.
Clear HTree index? no

Inode 1445497, i_size is 807660345911979152, should be 0.  Fix? no

Inode 1445497, i_blocks is 226835611961748, should be 0.  Fix? no

Inode 1445497 passes checks, but checksum does not match inode.  Fix? no

Inode 1445498 seems to contain garbage.  Clear? no

Inode 1445498 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445498 is in use, but has dtime set.  Fix? no

Inode 1445498 has imagic flag set.  Clear? no

Inode 1445498 has a extra size (58863) which is invalid
Fix? no

Inode 1445498 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445498 has an invalid root node.
Clear HTree index? no

Inode 1445498, i_size is 3597890145779093558, should be 0.  Fix? no

Inode 1445498, i_blocks is 114376193824259, should be 0.  Fix? no

Inode 1445498 passes checks, but checksum does not match inode.  Fix? no

Inode 1445499 seems to contain garbage.  Clear? no

Inode 1445499 seems to have inline data but extent flag is set.
Fix? no

Inode 1445499 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445499 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445499 is in use, but has dtime set.  Fix? no

Inode 1445499 has imagic flag set.  Clear? no

Inode 1445499 has a extra size (32194) which is invalid
Fix? no

Inode 1445499, i_size is 98167475422978941, should be 0.  Fix? no

Inode 1445499, i_blocks is 108922856597503, should be 0.  Fix? no

Inode 1445499 passes checks, but checksum does not match inode.  Fix? no

Inode 1445500 seems to contain garbage.  Clear? no

Inode 1445500 is in use, but has dtime set.  Fix? no

Inode 1445500 has imagic flag set.  Clear? no

Inode 1445500 has a extra size (37820) which is invalid
Fix? no

Inode 1445500 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445500, i_size is 11382194334749490044, should be 0.  Fix? no

Inode 1445500, i_blocks is 185057947390511, should be 0.  Fix? no

Inode 1445500 passes checks, but checksum does not match inode.  Fix? no

Inode 1445501 seems to contain garbage.  Clear? no

Inode 1445501 is in use, but has dtime set.  Fix? no

Inode 1445501 has imagic flag set.  Clear? no

Inode 1445501 has a extra size (53532) which is invalid
Fix? no

Inode 1445501 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445501 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445501 has an invalid root node.
Clear HTree index? no

Inode 1445501, i_size is 3284744134057677611, should be 0.  Fix? no

Inode 1445501, i_blocks is 147117222527488, should be 0.  Fix? no

Inode 1445501 passes checks, but checksum does not match inode.  Fix? no

Inode 1445502 seems to contain garbage.  Clear? no

Inode 1445502 is in use, but has dtime set.  Fix? no

Inode 1445502 has a extra size (19297) which is invalid
Fix? no

Inode 1445502 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445502 has corrupt extent header.  Clear inode? no

Inode 1445502, i_size is 14068936691664943963, should be 0.  Fix? no

Inode 1445502, i_blocks is 123082946978358, should be 0.  Fix? no

Inode 1445502 passes checks, but checksum does not match inode.  Fix? no

Inode 1445503 seems to contain garbage.  Clear? no

Inode 1445503 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Special (device/socket/fifo) file (inode 1445503) has extents
or inline-data flag set.  Clear? no

Inode 1445503 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445503 is in use, but has dtime set.  Fix? no

Inode 1445503 has a extra size (2037) which is invalid
Fix? no

Inode 1445503 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445503, i_size is 15303678105192228654, should be 0.  Fix? no

Inode 1445503, i_blocks is 231613845149030, should be 0.  Fix? no

Inode 1445503 passes checks, but checksum does not match inode.  Fix? no

Inode 1445504 seems to contain garbage.  Clear? no

Inode 1445504 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445504 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445504 is in use, but has dtime set.  Fix? no

Inode 1445504 has imagic flag set.  Clear? no

Inode 1445504 has a extra size (26069) which is invalid
Fix? no

Inode 1445504 has encrypt flag but no encryption extended attribute.
Clear flag? no

Inode 1445504 passes checks, but checksum does not match inode.  Fix? no

Inode 1445505 seems to contain garbage.  Clear? no

Inode 1445505 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445505 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445505 is in use, but has dtime set.  Fix? no

Inode 1445505 has imagic flag set.  Clear? no

Inode 1445505 has a extra size (10402) which is invalid
Fix? no

Inode 1445505, i_size is 15064529139542391244, should be 0.  Fix? no

Inode 1445505, i_blocks is 219654397518341, should be 0.  Fix? no

Inode 1445505 passes checks, but checksum does not match inode.  Fix? no

Inode 1445506 seems to contain garbage.  Clear? no

Inode 1445506 has the casefold flag set but is not a directory.  Clear flag? no

Directory ??? has the casefold flag, but the
casefold feature is not enabled.  Clear flag? no

Inode 1445506 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445506 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445506 is in use, but has dtime set.  Fix? no

Inode 1445506 has imagic flag set.  Clear? no

Inode 1445506 has a extra size (8273) which is invalid
Fix? no

Inode 1445506, i_size is 2712834718652506777, should be 0.  Fix? no

Inode 1445506, i_blocks is 177605813541400, should be 0.  Fix? no

Inode 1445506 passes checks, but checksum does not match inode.  Fix? no

Inode 1445507 seems to contain garbage.  Clear? no

Inode 1445507 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445507 is in use, but has dtime set.  Fix? no

Inode 1445507 has a extra size (59125) which is invalid
Fix? no

Inode 1445507 has encrypt flag but no encryption extended attribute.
Clear flag? no

HTREE directory inode 1445507 has an invalid root node.
Clear HTree index? no

Inode 1445507, i_size is 7231351303136584869, should be 0.  Fix? no

Inode 1445507, i_blocks is 110290005685788, should be 0.  Fix? no

Inode 1445507 passes checks, but checksum does not match inode.  Fix? no

Inode 1445508 seems to contain garbage.  Clear? no

Inode 1445508 has the ea_inode flag set but is not a regular file.  Clear flag? no

Inode 1445508 seems to have inline data but extent flag is set.
Fix? no

Inode 1445508 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445508 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445508 is in use, but has dtime set.  Fix? no

Inode 1445508 has a extra size (34327) which is invalid
Fix? no

Inode 1445508, i_size is 9178114816302797257, should be 0.  Fix? no

Inode 1445508, i_blocks is 129706788388368, should be 0.  Fix? no

Inode 1445508 passes checks, but checksum does not match inode.  Fix? no

Inode 1445509 seems to contain garbage.  Clear? no

Inode 1445509 seems to have inline data but extent flag is set.
Fix? no

Inode 1445509 has inline data and extent flags set but i_block contains junk.
Clear inode? no

Inode 1445509 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445509 is in use, but has dtime set.  Fix? no

Inode 1445509 has a extra size (34107) which is invalid
Fix? no

Inode 1445509 has corrupt extent header.  Clear inode? no

Inode 1445509 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1445509 has an invalid root node.
Clear HTree index? no

Inode 1445509, i_blocks is 59329083741334, should be 0.  Fix? no

Inode 1445509 passes checks, but checksum does not match inode.  Fix? no

Inode 1445510 seems to contain garbage.  Clear? no

Inode 1445510 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear? no

Inode 1445510 is in use, but has dtime set.  Fix? no

Inode 1445510 has a extra size (63135) which is invalid
Fix? no

Error while iterating over blocks in inode 1445510: Cannot iterate data blocks of an inode containing inline data

/dev/mapper/lvmg-arch2023: ********** WARNING: Filesystem still has errors **********

e2fsck: aborted

/dev/mapper/lvmg-arch2023: ********** WARNING: Filesystem still has errors **********

Anyone can make sense of this? should i try to fsck without -n on this?

My ideal course of action would be
- run fsck in a way that can be undone.
- try to mount RO
- copy data, or, undo fsck changes.

But i have no idea how to do this.

Last edited by gcb (2024-09-15 00:21:00)

frostschutz · 2024-09-14 18:34:39

If it's LVM, you can try: https://wiki.archlinux.org/title/LVM#Snapshots

Manual approach: https://raid.wiki.kernel.org/index.php/ … erlay_file

Experiments run on a copy-on-write overlay can be undone. But its also easy to make mistakes since the original device is still around after all. For example, if you mount by UUID - it might mount the wrong device.

Besides e2fsck you can also try your luck with debugfs.

Disable any overclock or boost if you doubt your ram or system stability...

Last edited by frostschutz (2024-09-14 18:36:20)

gcb · 2024-09-14 20:15:39

frostschutz wrote:

If it's LVM, you can try: https://wiki.archlinux.org/title/LVM#Snapshots
Manual approach: https://raid.wiki.kernel.org/index.php/ … erlay_file
Experiments run on a copy-on-write overlay can be undone. But its also easy to make mistakes since the original device is still around after all. For example, if you mount by UUID - it might mount the wrong device.
Besides e2fsck you can also try your luck with debugfs.
Disable any overclock or boost if you doubt your ram or system stability...

I've always run it with the most conservative performance options. Not that there are many options on laptop bios nowadays. I think i only disabled XMP and that was it.

Ok, so lots of good suggestions but everything new to me. Never found myself needing any of that. I actually go out of my way to use diposable qemu vms to not touch debugfs when testing kernel patches....

the lvm overlay sounds like a great idea. I can probably hold it in 32gb of ram easily, if that's an option... let's start reading....

gcb · 2024-09-14 22:06:45

Using the arch install media, i created an overlay file with 20G (have 32ram... assuming the installer ramdisk is autoalocate?)

i dmsetup the overlay on top of the physical partition and `cryptosetup open` the overlayed blockdevice /dev/mapper/fake-p5

then i `fsck` the lvm parition, and it seems to work. there's A LOT of output. Mostly pair of numbers scrolling by.

After all is done and fsck reports success i mount it, but then i start to get loopdevice write errors.

kermit63 · 2024-09-14 23:16:58

a bit off-topic, but since we're talking about being conservative, you really ought to invest some time thinking about back-ups.

Doing LVM snapshots even when there is no problem is good practice so you have something to fall back to when problem occurs. A major drawback of snapshots though is that it doesn't protect you from total disk failures since the back-up resides in the same disk/computer.

That's why I recommend porting such back-ups to other media (cloud, portable storage, other computers, etc.). Heck, you can replace the computer with a new one, and be up and running in less than an hour.

Another back-up tool that I use is partclone, which is available from the arch ISO. It works on block devices and has compression so the resulting backup takes less storage. In my experience, I'm getting about 50% reduction in storage space which is a big plus when porting to my somewhat limited cloud storage.

------

It's really a coin-toss when you get to the point where you have to do e2fsck to repair errors. Even if the repair succeeds, one can never shake off the feeling that there's still some underlying problems with the partition. In my case, when I get to this point, I usually just wipe the partition and reformat it, then restore from back-up for my peace of mind.

gcb · 2024-09-15 00:11:09

kermit63 wrote:

a bit off-topic, but since we're talking about being conservative, you really ought to invest some time thinking about back-ups.
Doing LVM snapshots even when there is no problem is good practice so you have something to fall back to when problem occurs. A major drawback of snapshots though is that it doesn't protect you from total disk failures since the back-up resides in the same disk/computer.
That's why I recommend porting such back-ups to other media (cloud, portable storage, other computers, etc.). Heck, you can replace the computer with a new one, and be up and running in less than an hour.
Another back-up tool that I use is partclone, which is available from the arch ISO. It works on block devices and has compression so the resulting backup takes less storage. In my experience, I'm getting about 50% reduction in storage space which is a big plus when porting to my somewhat limited cloud storage.
------
It's really a coin-toss when you get to the point where you have to do e2fsck to repair errors. Even if the repair succeeds, one can never shake off the feeling that there's still some underlying problems with the partition. In my case, when I get to this point, I usually just wipe the partition and reformat it, then restore from back-up for my peace of mind.

You are absolutely correct, and I do rsync over ssh everything weekly (and most of the actual work is pushed to remote repos all the time)

BUT... i still need to debug this. For two reasons, 1) i had session cookies to something that would be annoying to recover a password i forgot lol. and more importantly, 2) I can't replace the computer where I am right now. I need to investigate if it was just memory corruption or the SSD died because i will wipe this FS and reinstall, but I cannot change the hardware the best alternative to using the SSD I can do is run the OS from pendrives :skullemoji

good news is that it seems to have been the ram. i might try one stick at a time now, maybe i'm lucky and only one sodim is bad (i doubt it).

all partitions that were readonly at the time are OK. And the luks/lvm stuff wasn't impacted as far as i can tell. But I still cannot explain how it managed to do so much damage to the ext4 and the journal. All the damage was near the same pool of high inodes, which i'm interpreting as it being the more recent data only.

in case anyone finds themselves in this hell, here's my contribution (it's mostly concocted from the links shared by frostschutz earlier):

#!/bin/bash

# part to use "read-only" via an overlay
PART=/dev/nvme0n1p5

# some housecleaning for arch install image
setfont iso01-12x22
systemctl stop reflector.service

# everything that follow is from (just removed the `parallel` insanity)
# from https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

# N: Order number of the loop device
# <device-type>: Could be the followings
#    b: Block device (appropriate for loop devices)
#    c: Character device
#    p: (FIFO) Pipe device file
# <major>: Major device number. Typically "7" for loop devices
# <minor>: Minor device number. Typically match with "N"
#
#mknod /dev/loop<N> <device-type> <major> <minor>

# don't really need to, losetup will create one
#mknod -m 660 /dev/loop1 b 7 1

# size should be >=1% of the FS to recover? ideally if it can be the full size i guess...
truncate --size=500M overlay-p5
# NOTE: tried several sizes from 1G~16GB on the arch install ramdisk fs... when i mount the ext4 partition
#       i get loop errors. Maybe because it shows up as a loop file inside a loop mounted partition? not sure.
#       what worked was to use real device with 500M, and then mounting the target 'ro' just to be safe i won't use more than 500M.
#       and calling `sync` between every command that i expect to write to the overlay. it's kinda finicky for some reason
#       that i cannot explain yet.
# errors when using a overlay file in the loopback mounted ramdisk:
#  [ 1060.609611] loop: Write error at byte offset 267943936, length 4096.
#  [ 1060.610285] loop: Write error at byte offset 267939840, length 4096.
#  [ 1060.610754] I/O error, dev loop1, sector 523328 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.611243] loop: Write error at byte offset 267935744, length 4096.
#  [ 1060.612071] loop: Write error at byte offset 267931648, length 4096.
#  [ 1060.612884] loop: Write error at byte offset 267927552, length 4096.
#  [ 1060.613246] loop: Write error at byte offset 267923456, length 4096.
#  [ 1060.613601] loop: Write error at byte offset 267919360, length 4096.
#  [ 1060.613967] I/O error, dev loop1, sector 523320 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.613973] I/O error, dev loop1, sector 523312 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.613976] I/O error, dev loop1, sector 523304 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.613978] I/O error, dev loop1, sector 523296 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.613980] I/O error, dev loop1, sector 523288 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.615791] I/O error, dev loop1, sector 523280 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.616587] loop: Write error at byte offset 267915264, length 4096.
#  [ 1060.617691] loop: Write error at byte offset 267911168, length 4096.
#  [ 1060.618141] loop: Write error at byte offset 267907072, length 4096.
#  [ 1060.618594] I/O error, dev loop1, sector 523272 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.618980] I/O error, dev loop1, sector 523264 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.619354] I/O error, dev loop1, sector 523256 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
#  [ 1060.620869] device-mapper: snapshots: Invalidating snapshot: Error reading/writing.



size=$(blockdev --getsize $PART)
loop=$(losetup --find --show -- overlay-p5)
#loop=/dev/loop99

OVERLAY=/dev/mapper/p5
# not sure if we can pass full filename next... better keep what we
#  know works and pass only the mapper file name.
echo 0 $size snapshot $PART $loop P 8 | dmsetup create p5
# todo: use --table above

dmsetup status

# clean up:
# dmsetup remove p5
# rm overlay-p5
# losetup -d /dev/loop1

# mount overlay'ed partition:
#cryptsetup open /dev/mapper/p5 cryptroot
# time fsck.ext4 -y /dev/mapper/lvmg-arch2023 &> fsck.txt
#mount -o ro /dev/mapper/lvmg-arch2023 /mnt

btw, if anyone is reading this, the snapshot thing does not require lvm (well, it require the module because it is a lvm functionality, but the partition can be a raw one). I snapshoted the actual blockdevice/partition (which contains the LUKS encrypted data, which then contains the LVM, which then contains the FS i'm fsck-ing). So i guess it will work on any block device you need to work on but can't risk changes.

Last edited by gcb (2024-09-18 19:59:56)

seth · 2024-09-15 07:05:33

When the issue happens, everything "works" but it is as if I removed the ssd. There's no IO happening anywhere.

First and foremost assess the health status of the drive, https://wiki.archlinux.org/title/SMART (the overall status is useless, you'll have to look at the tabled values for pending or replaced sectors, uncorrectable errors etc.)
Then record the SSD model and see https://wiki.archlinux.org/title/Solid_ … leshooting - "overly optimistic" dipm support has shown up a lot in recent month, https://wiki.archlinux.org/title/Power_ … Management

Since it have full disk encryption

IO corruption on an encryted device can easily end up being fatal, you want to get this under control and not play russian roulette here.

Then dump a copy of the partition onto another disk as backup.
Then run fsck on the affected partition.
If it's your root partition, run

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files' | grep -v backup

to see whether there're package corruptions.
If you don't have hashes of other files that's the only thing you can validate.

Edit: sorry, missed

all partitions that were readonly at the time are OK. And the luks/lvm stuff wasn't impacted as far as i can tell. But I still cannot explain how it managed to do so much damage to the ext4 and the journal. All the damage was near the same pool of high inodes, which i'm interpreting as it being the more recent data only.

You still want to check the drive and data validity.

Last edited by seth (2024-09-15 07:06:45)

gcb · 2024-09-18 19:58:38

Excellent info as always. thanks!

I did cover all those a while ago. It's a quality enterprise samsung nvme. No *known* firmware issue. And nothing on logs or smart logged data.

And yes, i'm fully aware i'm playing russian roulette with full disk encryption. But my situation requires it. It gives me lots of worries but alas.

I'm slacking a lot to attach an external serial console and log what i get from there. Because I think it will be the only place that will generate any meaningful information, since all IO is gone when this happens. I cannot even read from other disks, like an attached pen drive (or I failed to manage to write to it on the few times i could try, because root was removed from under my feet) ... hum... i might try to point journald to a pen drive and see if I get something... i wonder.

But yeah, i'm just plowing trhu since the first two times it happened I noticed zero issues on the journal or data (compared to what was on disk against backup)... since then it happened some dozen times also without data corruption. This 48h event was the only one that managed to cause damage.

I think in the end this is all the ultimate endorsement to full disk encryption with luks on matters of stability

seth · 2024-09-18 20:21:44

So it's not been resolved w/ RAM adjustments?
nvme's are frequently troubled by APST, https://wiki.archlinux.org/title/Solid_ … leshooting

Even if you lose the root partition, keeping some "dmesg -w" should™ still post errors?

gcb · 2024-10-20 14:14:23

seth wrote:

So it's not been resolved w/ RAM adjustments?

Sorry, what you meant by that?

> nvme's are frequently troubled by APST

I'm running without the power management NOW (nvme_core.default_ps_max_latency_us=0)... I do have a samsung drive which is on the naughty list of SSD firmware support on linux. Seeing how much it impacts on battery.

> Even if you lose the root partition, keeping some "dmesg -w" should™ still post errors?

But ofcourse i used this laptop for months keeping dmesg (actually it was `journalctl --follow`) running on tty3, and the day i forget to set it, i the only one I get the error... Still unsure what is the actual failure.

Last edited by gcb (2024-10-20 14:14:56)

seth · 2024-10-20 18:23:42

You wrote:

good news is that it seems to have been the ram. i might try one stick at a time now, maybe i'm lucky and only one sodim is bad (i doubt it).

I'm running without the power management NOW

For clarification, that's post the most recent incident?

gcb · 2024-10-20 22:39:37

seth wrote:

You wrote:
good news is that it seems to have been the ram. i might try one stick at a time now, maybe i'm lucky and only one sodim is bad (i doubt it).
I'm running without the power management NOW
For clarification, that's post the most recent incident?

Nope. Recently I dropped full disk encryption (well, at least the part where the logs are stored and most of the 'fixed' base OS. only user data and portions of etc are under luks). Kept Both ram sticks as now i think it is the SSD firmware the likely culprit. And a couple boots ago dropped power management. (btw, `smartctl -c /dev/nvme0` before and after show the exact same info. including "Supported Power States". But i guess that's probing the driver, not what the kernel will use)

Still no crashes after adding nvme_core.default_ps_max_latency_us=0 but they don't show up very often anyway.

seth · 2024-10-21 06:01:39

Still no crashes after adding nvme_core.default_ps_max_latency_us=0

So you only added that *after* the last incident - that's what I meant.
Let's see…

Arch Linux

#1 2024-09-14 17:54:58

[SOLVED] can this ext4 FS be saved?

#2 2024-09-14 18:34:39

Re: [SOLVED] can this ext4 FS be saved?

#3 2024-09-14 20:15:39

Re: [SOLVED] can this ext4 FS be saved?

#4 2024-09-14 22:06:45

Re: [SOLVED] can this ext4 FS be saved?

#5 2024-09-14 23:16:58

Re: [SOLVED] can this ext4 FS be saved?

#6 2024-09-15 00:11:09

Re: [SOLVED] can this ext4 FS be saved?

#7 2024-09-15 07:05:33

Re: [SOLVED] can this ext4 FS be saved?

#8 2024-09-18 19:58:38

Re: [SOLVED] can this ext4 FS be saved?

#9 2024-09-18 20:21:44

Re: [SOLVED] can this ext4 FS be saved?

#10 2024-10-20 14:14:23

Re: [SOLVED] can this ext4 FS be saved?

#11 2024-10-20 18:23:42

Re: [SOLVED] can this ext4 FS be saved?

#12 2024-10-20 22:39:37

Re: [SOLVED] can this ext4 FS be saved?

#13 2024-10-21 06:01:39

Re: [SOLVED] can this ext4 FS be saved?

Board footer