[Resolved] Apparently, the new kernel slowed my nvme ssd

ratcheer · 2023-07-03 00:33:50

I just now installed the 6.4.1 kernel. I performed my usual filesystem maintenance (fstrim, btrfs balance, and btrfs scrub). I noticed that the speed of the scrub task was only 949 mb/sec. Its normal speed is somewhere from 2.8 to 3.2 gb/sec.

Are there any known issues about this?

 btrfs scrub status /mnt/nvme0n1p3/
UUID:             20db1fe2-60a4-4eb7-87ac-1953a55dda16
Scrub started:    Sun Jul  2 19:19:53 2023
Status:           finished
Duration:         0:00:51
Total to scrub:   47.28GiB
Rate:             948.61MiB/s
Error summary:    no errors found

inxi -F
System:
  Host: tux Kernel: 6.4.1-arch1-1 arch: x86_64 bits: 64 Console: pty pts/2 Distro: Arch Linux
Machine:
  Type: Desktop Mobo: ASUSTeK model: TUF GAMING X570-PLUS (WI-FI) v: Rev X.0x
    serial: 200771405807421 UEFI: American Megatrends v: 4602 date: 02/23/2023
CPU:
  Info: 12-core model: AMD Ryzen 9 3900X bits: 64 type: MT MCP cache: L2: 6 MiB
  Speed (MHz): avg: 2666 min/max: 2200/4672 cores: 1: 3800 2: 2200 3: 2200 4: 2200 5: 2200
    6: 3800 7: 2200 8: 3800 9: 2200 10: 2200 11: 3800 12: 2200 13: 3800 14: 2200 15: 2200 16: 2200
    17: 2200 18: 2200 19: 2200 20: 2200 21: 3800 22: 2200 23: 2200 24: 3800
Graphics:
  Device-1: NVIDIA TU104 [GeForce RTX 2060] driver: nvidia v: 535.54.03
  Display: server: X.org v: 1.21.1.8 driver: X: loaded: nvidia unloaded: modesetting gpu: nvidia
    tty: 273x63
  API: OpenGL Message: GL data unavailable in console and glxinfo missing.
Audio:
  Device-1: NVIDIA TU104 HD Audio driver: snd_hda_intel
  Device-2: AMD Starship/Matisse HD Audio driver: snd_hda_intel
  API: ALSA v: k6.4.1-arch1-1 status: kernel-api
Network:
  Device-1: Intel Wireless-AC 9260 driver: iwlwifi
  IF: wlan0 state: up mac: cc:d9:ac:3a:b4:9d
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169
  IF: enp5s0 state: down mac: 24:4b:fe:96:38:f9
Bluetooth:
  Device-1: N/A driver: btusb type: USB
  Report: rfkill ID: hci0 rfk-id: 0 state: down bt-service: disabled rfk-block: hardware: no
    software: no address: see --recommends
Drives:
  Local Storage: total: 7.73 TiB used: 378.62 GiB (4.8%)
  ID-1: /dev/nvme0n1 vendor: Western Digital model: WDBRPG0010BNC-WRSN size: 931.51 GiB
  ID-2: /dev/sda vendor: Samsung model: SSD 860 EVO 500GB size: 465.76 GiB
  ID-3: /dev/sdb vendor: Seagate model: ST2000DM008-2FR102 size: 1.82 TiB
  ID-4: /dev/sdc vendor: Western Digital model: WD50NDZW-11BGSS1 size: 4.55 TiB type: USB
Partition:
  ID-1: / size: 915.26 GiB used: 47.37 GiB (5.2%) fs: btrfs dev: /dev/nvme0n1p3
  ID-2: /boot size: 252 MiB used: 92.1 MiB (36.5%) fs: vfat dev: /dev/nvme0n1p1
  ID-3: /home size: 915.26 GiB used: 47.37 GiB (5.2%) fs: btrfs dev: /dev/nvme0n1p3
Swap:
  ID-1: swap-1 type: partition size: 16 GiB used: 0 KiB (0.0%) dev: /dev/nvme0n1p2
Sensors:
  System Temperatures: cpu: 27.5 C mobo: 26.0 C gpu: nvidia temp: 32 C
  Fan Speeds (RPM): fan-1: 847 fan-2: 1074 fan-3: 0 fan-4: 0 fan-5: 1002 fan-6: 0 fan-7: 782
Info:
  Processes: 407 Uptime: 23m Memory: available: 31.25 GiB used: 1.54 GiB (4.9%) Init: systemd
  Shell: Bash inxi: 3.3.27

Last edited by ratcheer (2023-07-04 13:00:12)

ratcheer · 2023-07-03 00:58:20

PS - The performance was normal the last time I ran the scrub, which was two days ago (Fri, June 30).

Last edited by ratcheer (2023-07-03 00:58:34)

ratcheer · 2023-07-03 01:14:32

Also, I have searched the Arch Bug tracker and do not find any reports that seem similar.

Later - And now I have searched the btrfs mail lists.

Last edited by ratcheer (2023-07-03 01:35:31)

seth · 2023-07-03 07:02:50

Did you try the LTS kernel and a synthetic IO benchmark to see whether it's a nvme related kernel regression?

ratcheer · 2023-07-03 12:42:23

I considered downgrading the kernel back to 6.3.9, but I haven't done it, yet. I will not upgrade anything else until I've decided whether to try that. It will also require an nvidia downgrade, I think.

Another thing I thought of is that there have been a lot of linux-firmware upgrades for the past few days. There was one Saturday morning and another one Sunday morning. Do those have anything to do with the nvme drive? I have browsed the tree, and I don't see anything that seems to relate to Western Digital or nvme.

When you say "synthetic IO benchmark", do you mean like hdparm?

PS - Ok, I installed and ran hdparm. This is the performance with the 6.4.1 kernel:

/dev/nvme0n1:
 Timing O_DIRECT cached reads:   3744 MB in  2.00 seconds = 1871.94 MB/sec
 Timing O_DIRECT disk reads: 9180 MB in  3.00 seconds = 3059.63 MB/sec

Last edited by ratcheer (2023-07-03 13:05:35)

seth · 2023-07-03 16:03:38

hdparm, dd - anything that just reads and writes the device will do.
You can install the LTS kernel parallel to the current one, if it's not the kernel I'd look at https://archlinux.org/packages/core/x86_64/btrfs-progs/ rather than the firmware packages first.

ratcheer · 2023-07-03 19:57:32

Well, yes, btrfs-progs was changed about a week ago (June 23 to be exact), but I'm fairly certain that I was getting my normal speeds after that. Still, I think I'll post on their mail list to see if anyone has any ideas.

Based on the hdparm output, it does appear that my SSD is still capable of full speed with this kernel: 3059.63 MB/sec is about what I am used to seeing from btrfs scrub.

Last edited by ratcheer (2023-07-03 21:51:50)

seth · 2023-07-03 20:13:25

btrfs is part of the kernel so to figure whether it's the tools or the filesystem you should test the current tools against the LTS kernel (and briefly hdparm on the LTS kernel to make sure to compare raw throughputs against each other and not some high-level scrubbing and low-level device IO)

ratcheer · 2023-07-03 21:51:05

Ok, thanks. I guess I'll set about figuring out how to do that (install LTS kernel parallel to the current one). Thanks.

ratcheer · 2023-07-04 13:03:57

I got this response from a btrfs developer. I am considering my issue as Resolved.

Well, the v6.4 kernel has introduced a new scrub implementation, which
has a completely different way of handling IOs.

In my initial tests, the new scrub should lead to less IOPS while higher
throughput.
But it doesn't look good at all for your case.

...

All the changes should lead to a better performance on slower disks, but
with your blazing fast devices, the csum verification may be a
bottleneck instead.

ratcheer · 2023-07-20 12:58:24

Followup:

My issue is leading to a near-future Linux kernel patch. My initial email to the btrfs development team is specifically cited as the initial report of the problem. But later, a handful of others reported the same problem.

Yay! Sometimes little guys can make a difference.

Arch Linux

#1 2023-07-03 00:33:50

[Resolved] Apparently, the new kernel slowed my nvme ssd

#2 2023-07-03 00:58:20

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#3 2023-07-03 01:14:32

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#4 2023-07-03 07:02:50

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#5 2023-07-03 12:42:23

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#6 2023-07-03 16:03:38

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#7 2023-07-03 19:57:32

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#8 2023-07-03 20:13:25

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#9 2023-07-03 21:51:05

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#10 2023-07-04 13:03:57

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

#11 2023-07-20 12:58:24

Re: [Resolved] Apparently, the new kernel slowed my nvme ssd

Board footer