regular BTRFS error killing my system

gtf21 · 2023-06-13 09:08:53

I had this once a few months ago, but in the last few days it has happened continually and made my system unusable. The symptom is that the system locks up, a few applications continue working for a bit but my WM disappears, other TTYs don't work, and eventually apps start dying.

If I switch TTY I see lots of the following errors:

[<number>]: BTRFS error (device dm-2: state EA): bdev /dev/mapper/MainVolGroup-root errs: wr 2, rd <some number>, flush 0, corrupt 0, gen 0

The constant is the device dm-2, state EA, bdev (...), and the wr number seems to be fixed but different each time -- the above example it was 2, the next time this happened it was 21.

I have tried booting from USB and using btrfs check and btrfs scrub and neither detect any errors. Somewhat thankfully, this only seems to be affecting the root partition, but I'm not sure what could be causing it. The increased frequency makes me worry it's an SSD problem.

Since btrfs check and btrfs scrub neither find any error nor fix the problem, I'm not sure what to do next -- I could reinstall the system but that feels like the wrong solution given that no errors are found.

seth · 2023-06-13 15:39:22

https://wiki.archlinux.org/title/SMART

If there's a parallel windows, see the 3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.

If this is an nvme, see https://wiki.archlinux.org/title/Solid_ … leshooting and disable APST and use the software IOMMU to see whether things stabilize.

The system shuts down/reboots cleanly w/ no "I yank the power plug" or "I push the power button for some seconds" involved?

gtf21 · 2023-06-13 15:47:58

1. No MS Windows anywhere on the machine
2. System does not shut down when I press the power button (quick press not long press) -- I get "Failed to execute shutdown binary", I assume because the filesystem is no longer "there"

Running from archiso I've been able to backup my system cleanly. I'll try the NVMe stuff to see if it helps and report back, thanks.

gtf21 · 2023-07-01 15:59:00

So, system was up for a good two weeks after this with the kernel options you suggested, but this morning when I tried to wake it up the system was totally broken, again suffering from "input/output errors". Unfortunately the logs were totally fried by this so I can't actually see what really happened (and the logs before the incident are not very interesting to interrogate -- just basic mail fetching service running as usual etc.).

gtf21 · 2023-07-01 16:31:41

Ok, and it just happened now as I was using the system, suddenly it locked up and I get lots of "read only file system", "disk I/O error" type errors. I can't run `dmesg` nor open `journalctl` to see what else is going on.

seth · 2023-07-01 18:47:50

Boot some live system so you don't rely on the built-in drives, check https://wiki.archlinux.org/title/SMART and see whether you can mount the drive - and possibly get errors for the attempt in the journal
Then please post your complete system journal for the boot:

sudo journalctl -b | curl -F 'f:1=<-' ix.io

Mr.Elendig · 2023-07-02 06:43:35

Time to backup any important data you haven't already asap

Arch Linux

#1 2023-06-13 09:08:53

regular BTRFS error killing my system

#2 2023-06-13 15:39:22

Re: regular BTRFS error killing my system

#3 2023-06-13 15:47:58

Re: regular BTRFS error killing my system

#4 2023-07-01 15:59:00

Re: regular BTRFS error killing my system

#5 2023-07-01 16:31:41

Re: regular BTRFS error killing my system

#6 2023-07-01 18:47:50

Re: regular BTRFS error killing my system

#7 2023-07-02 06:43:35

Re: regular BTRFS error killing my system

Board footer