[SOLVED] fsck checks f2fs partition at boot every time after an update

Kel05 · 2024-05-21 08:44:18

Yesterday, after a successful update, I rebooted my system and since then fcsk would just check and try to fix a "reported corruption" in my root partition every time I boot up my system. That update also had the new linux-zen kernel version (6.9.1), but either that isn't the source of the problem or whatever it did isn't reversible. Here's some info:

System

Kernel: linux-zen 6.8.9 (pre-update), linux-zen 6.9.1 (post-update)
File system: f2fs

What I already tried

Ran mkinitcpio -P. No change.
Used a different kernel (linux). No change.
Downgraded to the previous version (6.8.9). No change.
Ran fsck /dev/nvme0n1p2 from a Live boot image. Same result as on system boot (exit code: 0).
Set fs_passno to 0 in /etc/fstab. No change.
Prepend nofail to fs_mntops. No change.

/var/log/boot.log

Info: Fix the reported corruption.
Info: MKFS version
  "Linux version 6.7.2-arch1-2 (linux@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Wed, 31 Jan 2024 09:22:15 +0000"
Info: FSCK version
  from "Linux version 6.7.2-arch1-2 (linux@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Wed, 31 Jan 2024 09:22:15 +0000"
    to "Linux version 6.9.1-zen1-1-zen (linux-zen@archlinux) (gcc (GCC) 14.1.1 20240507, GNU ld (GNU Binutils) 2.42.0) #1 ZEN SMP PREEMPT_DYNAMIC Fri, 17 May 2024 16:56:18 +0000"
Info: superblock features = 0 : 
Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
Info: fs errors: invalid_blkaddr 
Info: Segments per section = 1
Info: Sections per zone = 1
Info: total FS sectors = 999188480 (487885 MB)
Info: CKPT version = 162ec42d
Info: checkpoint state = 45 :  crc compacted_summary unmount
[FSCK] Check node 1 / 830608 (0.00%)
[FSCK] Check node 83061 / 830608 (10.00%)
[FSCK] Check node 166121 / 830608 (20.00%)
[FSCK] Check node 249181 / 830608 (30.00%)
[FSCK] Check node 332241 / 830608 (40.00%)
[FSCK] Check node 415301 / 830608 (50.00%)
[FSCK] Check node 498361 / 830608 (60.00%)
[FSCK] Check node 581421 / 830608 (70.00%)
[FSCK] Check node 664481 / 830608 (80.00%)
[FSCK] Check node 747541 / 830608 (90.00%)
[FSCK] Check node 830601 / 830608 (100.00%)

[FSCK] Max image size: 83868 MB, Free space: 404016 MB
[FSCK] Unreachable nat entries                        [Ok..] [0x0]
[FSCK] SIT valid block bitmap checking                [Ok..]
[FSCK] Hard link checking for regular file            [Ok..] [0x12174]
[FSCK] valid_block_count matching with CP             [Ok..] [0x142e8d6]
[FSCK] valid_node_count matching with CP (de lookup)  [Ok..] [0xcac90]
[FSCK] valid_node_count matching with CP (nat lookup) [Ok..] [0xcac90]
[FSCK] valid_inode_count matched with CP              [Ok..] [0xc7224]
[FSCK] free segment_count matched with CP             [Ok..] [0x303a7]
[FSCK] next block offset is free                      [Ok..]
[FSCK] fixing SIT types
[FSCK] other corrupted bugs                           [Ok..]
[update_superblock: 765] Info: Done to update superblock

Done: 55.397368 secs

/etc/fstab

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p2 LABEL=ROOT
UUID=aab20f54-1a25-433b-aa1c-e77597bc3cdd	/         	f2fs      	rw,relatime,lazytime,background_gc=on,nogc_merge,discard,discard_unit=block,no_heap,inline_xattr,inline_data,inline_dentry,flush_merge,barrier,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,checkpoint_merge,fsync_mode=posix,memory=normal,errors=continue	0 1

# /dev/zram0
/dev/zram0 		none 		swap 		defaults,pri=100   0 0

# /dev/nvme0n1p1 LABEL=BOOT
UUID=23F6-3F72      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

/etc/mkinitcpio.conf

MODULES=()
BINARIES=()
FILES=()

HOOKS=(base udev plymouth autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)

Any kind of help will be very appreciated cause I don't know what to do other than reinstalling Arch. This isn't a severe problem as it doesn't impact the system's integrity, but it still added a minute to the boot time, where before it took mere seconds to boot.

Last edited by Kel05 (2024-05-21 22:30:45)

birdspider · 2024-05-21 20:08:40

I have exactly the same issue* (invalid_blkaddr, fsck does not help, first mount after boot takes ~2.5min), and here (forum.archlinux.de) someone else also has that problem.

EDIT: *) on a non-root drive, so the mount actually happens, while the system is up - only delayed

---

Analysis: dd example taken from https://lore.kernel.org/linux-f2fs-deve … l.org/T/#u

1. dd if=/dev/zero of=test.img count=100 bs=1M
2. mkfs.f2fs -f ./test.img
3. fsck.f2fs test.img  | grep errors # nothing
4. mount -t f2fs ./test.img ./mnt
5. umount ./test.img
6. fsck.f2fs test.img  | grep errors 
   # Info: fs errors: invalid_blkaddr

---

filed an issue upstream: https://bugzilla.kernel.org/show_bug.cgi?id=218870

Last edited by birdspider (2024-05-21 21:04:29)

Kel05 · 2024-05-21 21:18:21

I see. From the forum link you sent they said that rebooting two times after the kernel downgrade resolved the problem. I replicated that and it actually worked!

In the end it was a kernel issue, I just wasn't patient enough and only rebooted once, lol. I guess, we'll need to wait for a kernel update to resolve the issue. I will just hold the package 'till then.

Anyways, thanks for the help!

birdspider · 2024-05-21 21:31:10

Kel05 wrote:

I will just hold the package 'till then.

Do you mean, when you downgrade to 6.8.9 PLUS let fsck finish once, the issue remains mitigated?

---

From the forum:

Ab jetzt bootet der PC ganz normal schnell durch.

I missed the part where he downgraded.

Last edited by birdspider (2024-05-21 21:33:12)

Kel05 · 2024-05-21 22:27:18

birdspider wrote:

Do you mean, when you downgrade to 6.8.9 PLUS let fsck finish once, the issue remains mitigated?

Exactly.

Erwin Iosef · 2024-05-22 10:14:38

I have the same problem. Tried it with the lts 6.4.12 kernel and no change.

Kel05 · 2024-05-22 15:12:44

That's strange, but why use such an old version? Where you on Linux lts too before updating? Then your issue might be something else, as the source of my problem was a bug specifically found on version 6.9.x.

Erwin Iosef · 2024-05-23 17:28:33

No, I have the latest zen kernel too and I updated my system yesterday and I discovered the problem while using said kernel. Who knows? It could be both. Or something else.

b83c · 2024-05-25 00:22:40

Faced the same issue as described, but switching from mkinitpcio to booster seemed to have circumvented the annoying fsck step.

hunabku · 2024-05-25 08:00:14

Same probkem for me with dracut.

GTP · 2024-05-25 16:21:13

I had the same problem, both for my home and root partitions. But I just tried running 'fsck.f2fs -f' on the affected partitions from a live image and it fixed the issue without downgrading the kernel. I have an LVM on LUKS setup.

EDIT: I have to take it back. It only worked for the first boot after running fsck, the next boot it happened again.

Last edited by GTP (2024-05-25 17:16:43)

tomz17 · 2024-05-27 16:36:44

Same here: I believe it's this: https://bugzilla.kernel.org/show_bug.cgi?id=218770 and 6.9.2 will have the fix

Edit: does NOT appear to be fixed in 6.9.2-arch1-1 even after multiple reboots, the link to the gentoo post in German above also mentions that 6.9.2 did not fix it for them

Edit: downgrading to 6.6.32-1-lts and rebooting twice did work

Last edited by tomz17 (2024-05-27 17:19:17)

jpegxguy · 2024-05-27 20:23:42

Damn, I was just going to ask if anyone tried the latest kernel (6.9.2)
Seems like I shouldn't bother yet

sosonok · 2024-05-30 14:25:26

https://bugzilla.kernel.org/show_bug.cgi?id=218770#c15

correction; 6.9.3-rc1 has the fix.

birdspider · 2024-05-31 23:32:42

jpegxguy wrote:

Damn, I was just going to ask if anyone tried the latest kernel (6.9.2)
Seems like I shouldn't bother yet

6.9.3 in [core-testing] has the fix, and should land soon. I tried it - and the fix works.

Erwin Iosef · 2024-06-01 12:03:43

That's good because the latest kernel of yesterday (don't know about now) also didn't work so far.
What about the lts kernel. Does it have a fix as well?

Erwin Iosef · 2024-06-02 16:43:23

Tried 6 9.3 today. Didn't work.
Anyone else?

jpegxguy · 2024-06-05 21:05:35

Erwin Iosef wrote:

Tried 6 9.3 today. Didn't work.
Anyone else?

Keep in mind you might need to reboot twice to see it solved.

I'l ltry with 6.9.3 now

UPDATE working fine for me on 6.9.3 after 2 boots

Last edited by jpegxguy (2024-06-05 21:35:23)

Erwin Iosef · 2024-06-07 17:45:23

Yes I tried it too. And goddamn it worked!
Thanks for the tip.

Erwin Iosef · 2024-06-09 17:05:01

It happened again after an update today.
After the reboot practice it went away.

Arch Linux

#1 2024-05-21 08:44:18