IO errors after recent update

theramatrix · 2025-11-17 01:35:59

I've got an ASUS laptop (S13 OLED, UM5302) that I've been using for ~2 years, after a recent software update I've been getting a bunch of crashes. Trying to run terminal commands result in 'input output error'.

What I've tried:
- Running fsck
- Running smartctl
- Switching to LTS kernel
- Reinstalling
- Swapping out the hard drive
- Trying a different distro
- Trying Windows (Alas didn't crash, annoyingly, ruling out simple hardware failure)

journalctl -b log: https://pastebin.com/Nxr8j0JS
dmesg log: https://pastebin.com/c3fGsxj8

Help appreciated.

Last edited by theramatrix (2025-11-17 05:58:40)

seth · 2025-11-17 15:58:30

The journal doesn't seem to include any IO errors nor other "crashes"
Generically see https://wiki.archlinux.org/title/Solid_ … leshooting

pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 iommu=soft

Trying Windows

In case you're dual booting, see the 3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.

theramatrix · 2025-11-18 04:16:11

So I got the feeling the IO errors meant the logs became unwritable once they started happening, but it occured to me I could try booting off the Arch install ISO with the drive mounted. Here's what I got:

dmesg:

[ 2834.887074] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 2834.887089] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 2834.887094] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
[ 2834.920687] nvme 0000:02:00.0: enabling device (0000 -> 0002)
[ 2834.920883] nvme nvme0: Disabling device after reset failure: -19
[ 2834.933270] Buffer I/O error on dev nvme0n1p2, logical block 29, lost async page write
[ 2834.933342] EXT4-fs (nvme0n1p2): shut down requested (2)
[ 2834.933392] Buffer I/O error on dev nvme0n1p2, logical block 66, lost async page write
[ 2834.933479] Aborting journal on device nvme0n1p2-8.
[ 2834.933513] Buffer I/O error on dev nvme0n1p2, logical block 9567, lost async page write
[ 2834.933611] Buffer I/O error on dev nvme0n1p2, logical block 59768848, lost async page write
[ 2834.933612] Buffer I/O error on dev nvme0n1p2, logical block 121667584, lost sync page write
[ 2834.933692] Buffer I/O error on dev nvme0n1p2, logical block 59768864, lost async page write
[ 2834.933758] JBD2: I/O error when updating journal superblock for nvme0n1p2-8.
[ 2834.933848] Buffer I/O error on dev nvme0n1p2, logical block 136314883, lost async page write
[ 2834.933996] Buffer I/O error on dev nvme0n1p2, logical block 136314885, lost async page write
[ 2834.934077] Buffer I/O error on dev nvme0n1p2, logical block 243927573, lost async page write

journalctl:

Nov 18 04:09:55 archiso kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
Nov 18 04:09:55 archiso kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
Nov 18 04:09:55 archiso kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Nov 18 04:09:55 archiso kernel: nvme 0000:02:00.0: enabling device (0000 -> 0002)
Nov 18 04:09:55 archiso kernel: nvme nvme0: Disabling device after reset failure: -19
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 29, lost async page write
Nov 18 04:09:55 archiso kernel: EXT4-fs (nvme0n1p2): shut down requested (2)
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 66, lost async page write
Nov 18 04:09:55 archiso kernel: Aborting journal on device nvme0n1p2-8.
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 9567, lost async page write
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 59768848, lost async page write
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 121667584, lost sync page write
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 59768864, lost async page write
Nov 18 04:09:55 archiso kernel: JBD2: I/O error when updating journal superblock for nvme0n1p2-8.
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 136314883, lost async page write
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 136314885, lost async page write
Nov 18 04:09:55 archiso kernel: Buffer I/O error on dev nvme0n1p2, logical block 243927573, lost async page write

pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 iommu=soft

Thanks for the suggestion, but no dice.

In case you're dual booting

I'm not, I just figured since I was reinstalling I might as well see if Windows had the bug.

Last edited by theramatrix (2025-11-19 01:39:56)

seth · 2025-11-18 09:05:17

Please use [code][/code] tags. Edit your post in this regard.

Thanks for the suggestion, but no dice.

Nov 18 04:09:55 archiso kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
Nov 18 04:09:55 archiso kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
Nov 18 04:09:55 archiso kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug

Does

cat /proc/cmdline

show that you're indeed booting w/ those parameters?
Please post the entire system journal covering such failure:

sudo journalctl -b | curl -F 'file=@-' 0x0.st

will upload it to 0x0.st and hand you a link to share.

theramatrix · 2025-11-19 01:44:13

Does
cat /proc/cmdline
show that you're indeed booting w/ those parameters?

Yes, I can confirm that command returns the kernel parameters as expected.

Please post the entire system journal covering such failure:
sudo journalctl -b | curl -F 'file=@-' 0x0.st
will upload it to 0x0.st and hand you a link to share.

https://0x0.st/Kf6J.txt

seth · 2025-11-19 08:35:30

Nov 19 01:34:59 archiso kernel: DMI: ASUSTeK COMPUTER INC. Zenbook UM5302LA_UM5302LA/UM5302LA, BIOS UM5302LA.302 07/27/2023
Nov 19 01:34:59 archiso kernel: ACPI: AC: AC Adapter [ACAD] (on-line)

The nvme is stock? You didn't add that after market?
Do you get the same errors w/ https://grml.org/ (for a different kernel, config, userspace)?

theramatrix · 2025-11-20 03:41:34

seth wrote:

The nvme is stock? You didn't add that after market?

Well, it's aftermarket *now*, but the issues started happening with the original SSD that came with the factory.

seth wrote:

Do you get the same errors w/ https://grml.org/ (for a different kernel, config, userspace)?

I've been running it for about an hour (longer than any period it took for the IO errors to start happening) and I don't seem to. Keeping my laptop on and running grml, will report if the IO errors happen.

Last edited by theramatrix (2025-11-20 03:42:33)

seth · 2025-11-20 08:20:22

I don't seem to. Keeping my laptop on and running grml

Ftr, you are accessing the drive from there, it's not just hanging around?

- Switching to LTS kernel
- Swapping out the hard drive
- Trying a different distro
- Trying Windows (Alas didn't crash, annoyingly, ruling out simple hardware failure)

1. what other distro? (apparently not grml )
2. try to downgrade to an older kernel (any bug might have been backported into the LTS one) AND firmware
Iff it's not the HW at all the problem will come from those two and be w/ the bus (since you switched the drive itself as well)

theramatrix · 2025-11-22 08:51:33

Ftr, you are accessing the drive from there, it's not just hanging around?

Yes, I ran

mkdir

and

rmdir

a bunch of times to test

1. what other distro? (apparently not grml )
2. try to downgrade to an older kernel (any bug might have been backported into the LTS one) AND firmware
Iff it's not the HW at all the problem will come from those two and be w/ the bus (since you switched the drive itself as well)

It was a copy of Nobara I had lying around. I don't get it though - Nobara offered the option to boot a 6.17 kernel and the previous 6.16 kernel. I tried both and the IO errors still happened (as well as on a fresh install booting with v6.17.6 like you suggested) - if the errors can be observed on 6.16 then the bug must have been introduced at least that far back and exist throughout 6.17.x? I hadn't changed what packages were installed nor updated my BIOS; why is it only rearing its head now?

seth · 2025-11-22 14:14:25

a bunch of times to test

Try to actually read (or write) some bigly-ish file there, "mkdir foo; rmdir foo" will happen in the VM and especially if you do this in quick™ succession never synced, ie. there's effectively no IO on the drive.

theramatrix · 2025-11-22 23:05:12

Okay, I booted back up grml, copied a Windows 10 ISO to the problematic drive and ran sha256sum on it, which returned the correct hash. Then I waited a few minutes and ran sha256sum again. Still no issue.

seth · 2025-11-22 23:28:35

I hadn't changed what packages were installed nor updated my BIOS; why is it only rearing its head now?

after a recent software update I've been getting a bunch of crashes.

seth wrote:

try to downgrade to an older kernel (any bug might have been backported into the LTS one) AND firmware

Check /var/log/pacman.log for what versions got changed, roll back to the previous ones, see whether the issue remains.
If the errors reliably go away we can start looking for the critical change - if not we'll have to look for pot. mitigating factors of grml.

nb

Then I waited a few minutes and ran sha256sum again. Still no issue.

sudo sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

will work better than waiting any random amount of minutes where nothing happens

theramatrix · 2025-11-23 01:45:15

I couldn't check /var/log directly since I, er, wiped my drive troubleshooting, so I went with installing old snapshots from archive.archlinux.org

The snapshots from Oct 1 and Nov 1 was no good but the one from Sep 1 seemed to be stable
So here's the output of `pacman -Qu` with the Sep 1 snapshot installed and the /etc/pacman.d/mirrorlist pointing to Oct 1 snapshot

archlinux-keyring 20250807.1-1 -> 20250929-1
ca-certificates-mozilla 3.115.1-1 -> 3.116-1
coreutils 9.7-1 -> 9.8-2
curl 8.15.0-1 -> 8.16.0-1
device-mapper 2.03.34-1 -> 2.03.35-1
ell 0.79-1 -> 0.80-1
expat 2.7.1-1 -> 2.7.3-1
glib2 2.84.4-2 -> 2.86.0-2
gpgme 2.0.0-1 -> 2.0.1-1
hwdata 0.398-1 -> 0.399-1
iproute2 6.16.0-2 -> 6.17.0-1
iwd 3.9-1 -> 3.10-1
libffi 3.5.1-1 -> 3.5.2-1
libnghttp2 1.66.0-1 -> 1.67.1-1
libnghttp3 1.11.0-1 -> 1.12.0-1
libp11-kit 0.25.5-1 -> 0.25.9-1
libsysprof-capture 48.1-2 -> 49.0-1
libtool 2.5.4+r62+g6389dce8-1 -> 2.6.0-1
libxml2 2.14.5-1 -> 2.15.0-1
linux 6.16.4.arch1-1 -> 6.16.8.arch3-1
linux-firmware 20250808-1 -> 20250917-1
linux-firmware-amdgpu 20250808-1 -> 20250917-1
linux-firmware-atheros 20250808-1 -> 20250917-1
linux-firmware-broadcom 20250808-1 -> 20250917-1
linux-firmware-cirrus 20250808-1 -> 20250917-1
linux-firmware-intel 20250808-1 -> 20250917-1
linux-firmware-mediatek 20250808-1 -> 20250917-1
linux-firmware-nvidia 20250808-1 -> 20250917-1
linux-firmware-other 20250808-1 -> 20250917-1
linux-firmware-radeon 20250808-1 -> 20250917-1
linux-firmware-realtek 20250808-1 -> 20250917-1
linux-firmware-whence 20250808-1 -> 20250917-1
linux-headers 6.16.4.arch1-1 -> 6.16.8.arch3-1
mkinitcpio 39.2-3 -> 39.2-5
openssl 3.5.2-1 -> 3.5.3-1
p11-kit 0.25.5-1 -> 0.25.9-1
pinentry 1.3.2-1 -> 1.3.2-2
sqlite 3.50.4-1 -> 3.50.4-2
systemd 257.8-2 -> 258-4
systemd-libs 257.8-2 -> 258-4
systemd-sysvcompat 257.8-2 -> 258-4
util-linux 2.41.1-1 -> 2.41.2-1
util-linux-libs 2.41.1-1 -> 2.41.2-1

Quick edit inb4 it causes confusion: No, I hadn't not run a system update since October. I don't know why these IO errors are happening now but not during actual October last month.

Last edited by theramatrix (2025-11-23 01:58:44)

seth · 2025-11-23 09:22:19

linux 6.16.4.arch1-1 -> 6.16.8.arch3-1

Slet's see whether updating to any of

pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.16.5.arch1-1-x86_64.pkg.tar.zst
pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.16.6.arch1-1-x86_64.pkg.tar.zst
pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.16.7.arch1-1-x86_64.pkg.tar.zst

breaks it again.

theramatrix · 2025-11-29 04:36:11

Well, I'm annoyed, because I tried all the versions and the answer is no. NGL I'm not exactly eager to test a potential dozen or more permutations of packages to figure out which combo is causing the issue.

seth · 2025-11-29 09:25:08

You can incrementally update using https://wiki.archlinux.org/title/Arch_L … cific_date

But realistically it's the kernel, the firmware, systemd (udev rules - don't change systemd w/o systemd-libs or vv) or maybe device-mapper (do you even use that?)

theramatrix · 2025-12-13 10:52:16

Okay - I found some free time to continue troubleshooting and I think it's indeed one of the linux-firmware pkgs. Do I need to diff the exact code change or just submit a bug report (where if so)? Getting out of my depth here tbh...

seth · 2025-12-13 12:09:15

just submit a bug report (where if so)?

Do you know which firmware package and which versions get loaded in the good and the bad case?

theramatrix · 2025-12-13 13:43:13

seth wrote:

just submit a bug report (where if so)?
Do you know which firmware package and which versions get loaded in the good and the bad case?

No, but I'm guessing it's linux-firmware itself since the others, judging by the name, are network and GPU stuff? The good one is 20250808-1 and the bad one is 20250917-1

seth · 2025-12-13 17:27:30

https://archlinux.org/packages/core/any/linux-firmware/ is a metapackage that installs the other firmware packages, it does nothing by itself.
You might want to check which of the firmwares you actually need and then test the remaining ones individually.
It's probably linux-firmware-amdgpu, linux-firmware-cirrus or linux-firmware-mediatek

theramatrix · 2025-12-16 12:46:45

It was (is?), indeed, linux-firmware-amdgpu. Downgrading that back to the 20250808 version solved it; laptop's been sitting for a solid day with no issues.

Now do I have to diff and figure out the exact problematic code change or submit a bug report? Where if so?

Banjo536 · 2025-12-17 10:38:48

I figured out the IO errors were caused by linux-firmware-amdgpu. I downgraded it to version 20250808-1 and my laptop has been stable since. I will submit a bug report to the Arch Linux bug tracker so they can investigate the problematic update.

seth · 2025-12-17 14:00:59

Don't, file a bug at https://gitlab.freedesktop.org/drm/amd/-/issues/
The firmware is a blackbox - AMD will have to fix that.

Arch Linux

#1 2025-11-17 01:35:59

IO errors after recent update

#2 2025-11-17 15:58:30

Re: IO errors after recent update

#3 2025-11-18 04:16:11

Re: IO errors after recent update

#4 2025-11-18 09:05:17

Re: IO errors after recent update

#5 2025-11-19 01:44:13

Re: IO errors after recent update

#6 2025-11-19 08:35:30

Re: IO errors after recent update

#7 2025-11-20 03:41:34

Re: IO errors after recent update

#8 2025-11-20 08:20:22

Re: IO errors after recent update

#9 2025-11-22 08:51:33

Re: IO errors after recent update

#10 2025-11-22 14:14:25

Re: IO errors after recent update

#11 2025-11-22 23:05:12

Re: IO errors after recent update

#12 2025-11-22 23:28:35

Re: IO errors after recent update

#13 2025-11-23 01:45:15

Re: IO errors after recent update

#14 2025-11-23 09:22:19

Re: IO errors after recent update

#15 2025-11-29 04:36:11

Re: IO errors after recent update

#16 2025-11-29 09:25:08

Re: IO errors after recent update

#17 2025-12-13 10:52:16

Re: IO errors after recent update

#18 2025-12-13 12:09:15

Re: IO errors after recent update

#19 2025-12-13 13:43:13

Re: IO errors after recent update

#20 2025-12-13 17:27:30

Re: IO errors after recent update

#21 2025-12-16 12:46:45

Re: IO errors after recent update

#22 2025-12-17 10:38:48

Re: IO errors after recent update

#23 2025-12-17 14:00:59

Re: IO errors after recent update

Board footer