Broken /proc/diskstats counter in 7.1.3.arch1-3?

disarmer · 2026-07-15 00:32:00

Hi everyone!

After a system update (kernel upgraded from 7.0.8.arch1‑1 to 7.1.3.arch1‑3), I noticed that the disk statistics counters in /proc/diskstats are misbehaving. The following fields show sudden spikes:
11 – time spent writing (ms)
14 – weighted time spent doing I/Os (ms)

The increase between two consecutive measurements (15‑second interval) can be tens of thousands of seconds, even though there is no significant I/O load on the machine (it's a regular desktop). At the same time, the IOPS and written bytes counters grow smoothly.

I collect metrics from /proc/diskstats for my NVMe device (nvme0n1) every 15 seconds. According to the kernel documentation (https://www.kernel.org/doc/Documentatio … -diskstats), field 11 is the time spent writing to the disk. I convert it to seconds and push it to VictoriaMetrics.

Before today's update, everything worked perfectly – both counters increased gradually by milliseconds. After the reboot, however, I'm seeing sharp spikes (14 times in the last 6 hours).

My setup uses btrfs on top of a LUKS‑encrypted partition, but I don't think that should matter, since the counters reflect activity of the raw device (nvme0n1).

Can anyone reproduce this behaviour on their system? Here's a one‑liner that extracts field 11 from the first line of /proc/diskstats and converts it to seconds:

head -1 /proc/diskstats | perl -nwE 'say +(split / +/, $_)[11]/1000;'

Does your counter also jump erratically from time to time?

I've double‑checked my monitoring script – no errors there. I'm attaching a screenshot of the graph and a dump of the metric in InfluxDB format (where the time field spikes twice):

          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7451575296,merged=72879,ops=416960,time=128466.228 1784071265000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7481259008,merged=72951,ops=417738,time=128466.832 1784071280000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7482045440,merged=72951,ops=417792,time=128466.907 1784071295000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7491482624,merged=72961,ops=418447,time=150373.242 1784071310000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7493817344,merged=72962,ops=418545,time=150373.36 1784071325000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7510496256,merged=73226,ops=419485,time=150373.988 1784071340000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7512732672,merged=73226,ops=419681,time=150374.09 1784071355000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7517037568,merged=73246,ops=419985,time=150374.37 1784071370000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7524545536,merged=73354,ops=420338,time=150374.636 1784071385000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7528145920,merged=73363,ops=420601,time=150374.902 1784071400000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7534199808,merged=73409,ops=420967,time=150375.105 1784071415000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7535404032,merged=73409,ops=421063,time=150375.228 1784071430000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7543198720,merged=73467,ops=421498,time=150375.587 1784071445000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7546741760,merged=73467,ops=421788,time=150375.748 1784071460000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7557157888,merged=73580,ops=422433,time=172437.053 1784071475000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7558738944,merged=73580,ops=422569,time=172437.151 1784071490000000000',
          'diskstats,dev=nvme0n1,host=desktop,type=write bytes=7566795776,merged=73611,ops=422999,time=172437.644 1784071505000000000',

Last edited by disarmer (2026-07-15 08:51:59)

seth · 2026-07-15 07:38:02

Please replace the oversized image w/ a link (the board has 250x250 max rule to keep mousewheels cool)

If the overall load hasn't changed and just transitioned from a curve to a staircase I'd assume more aggressive caching (maybe due to APST/power saving keeping the drive idle)?

disarmer · 2026-07-15 09:49:32

Thanks, replaced with preview

I'm not sure APST (or any power‑management feature) should affect these counters – that would break pretty much every monitoring tool out there.

Also, the counter didn't simply turn into a staircase pattern. Between the spikes it still grows smoothly, just like before – you just can't see that on the scale of the spikes. I'm attaching a 12‑hour graph view, along with a segment where no spikes occurred. In that spike‑free part, the counter increased by about 400 seconds over 5.5hours – which looks perfectly plausible. However, with the spikes included, the total growth reaches 260000 seconds over just 3hours – that simply cannot be right.

seth · 2026-07-15 14:20:00

260000 seconds over just 3hours – that simply cannot be right

I still need to watch TENET

260000 is ~3d and the general uptick would be by a factor of ~1192 so maybe 1k?

There's no switch to microseconds recorded at https://www.kernel.org/doc/Documentatio … ostats.rst - and that would also be nuts and not fitting the intermittent nature.

Any kind of pattern to stuff that's happening (in userspace) during those events (notably wrt the CPU, but eg also S3 cycles)?

disarmer · 2026-07-15 18:37:33

The computer was on all the time, without hibernation or powersave. The load was almost always light: browsing, games, the terminal, maybe watching YouTube videos. There was almost no disk activity, mostly discard

I've only noticed a couple of patterns so far:
1. When the system was completely idle (4:00 AM - 10:30 AM), there were far fewer spikes.
2. The higher the uptime, the larger the spikes. Here are the graphs: uptime is about a day, the first graph shows the value in seconds, the second shows the delta. It's noticeable that the delta grows almost linearly

disarmer · 2026-07-16 22:41:00

A few more observations:
1. The problem is reproducible after updating from 7.1.3.arch1-3 to 7.1.3.arch2-1 and rebooting.
2. CPU load doesn't affect the reproducibility rate (I stressed all cores for half an hour).
3. The "Field 4 -- # of milliseconds spent reading" field also spikes, although much less frequently (only three spikes in two days).

Arch Linux

#1 2026-07-15 00:32:00

Broken /proc/diskstats counter in 7.1.3.arch1-3?

#2 2026-07-15 07:38:02

Re: Broken /proc/diskstats counter in 7.1.3.arch1-3?

#3 2026-07-15 09:49:32

Re: Broken /proc/diskstats counter in 7.1.3.arch1-3?

#4 2026-07-15 14:20:00

Re: Broken /proc/diskstats counter in 7.1.3.arch1-3?

#5 2026-07-15 18:37:33

Re: Broken /proc/diskstats counter in 7.1.3.arch1-3?

#6 2026-07-16 22:41:00

Re: Broken /proc/diskstats counter in 7.1.3.arch1-3?

Board footer