[SOLVED] Random I/O freezes

Papi · 2018-11-10 10:12:26

Hi,
I am using ArchLinux since a few years now. I have problem with random I/O freezes since end of the September. I've tried to downgrade kernel version, but it didn't help.

Error description
* system normally works, but I can't load page in browser, open new terminal window and some apps become transparent
* display works smoothly and responsively
* music and videos normally plays when it's cached in RAM, but I can't load next song or watch next part of video when it needs to be downloaded from internet (Spotify, YouTube)
* dmesg and journalctl is empty, no errors
* ioping shows normal values when I/O is freezed (about 200μs (best) - 2ms (worst))
* dstat shows missed x ticks very often
* dstat, iostat and iotop shows very high peak of I/O usage when it stops to be freezed

Possible causes
It happens very randomly. I/O freezes every 2 minutes or 2 hours. One freeze took usually from few seconds to half a minute.
Usually it happens immediately after boot when I start a lot of apps at once (Chrome, Telegram, Slack, Caprine, Spotify, Thunderbird)

Laptop information
Model: Dell Precision 3530
CPU: Intel Core Processor i5-8400H (4 Core, 8MB Cache, 2.50GHz, 4.2GHz Turbo, 35W), vPro
GPU: Nvidia Quadro P600 w/ 4GB GDDR5 (disabled), Intel® UHD Graphics 630
RAM: 16GB, DDR4-2666MHz SDRAM, 2 DIMMS, Non-ECC
SSD: Kingston HyperX Savage 480GB, 2.5", SHSS37A/480G

System information
* Kernel: 4.18.16-arch1-1-ARCH
* Filesystem: btrfs

I've also tried to benchmark my disk using dd if=/dev/zero of=file bs=4M count=10000 status=progress
* Blue - normal system
* Orange - Arch Linux from USB with mounted my btrfs filesystem

https://i.imgur.com/2YiVPnw.png
(X-axis - record from dstat, Y-axis - MB/s)

Peaks in a blue line happens after freeze.

Mod note: Please don't embed large images in your posts, use URLs or linked thumbnails -- https://wiki.archlinux.org/index.php/Co … s_and_code -- WorMzy

Last edited by Papi (2018-11-15 07:59:56)

WorMzy · 2018-11-10 10:47:36

Hi Papi, welcome to the Arch Linux forums.

Please provide a full journal from an affected session (e.g. use 'journalctl -b > journal', and then a pastebin client to upload it). The output from

smartctl -a /dev/sda

(assuming your SSD is enumerated as sda)

and

btrfs filesystem show
btrfs device stats /

Papi · 2018-11-10 11:57:57

journalctl -b: https://ptpb.pw/5fbe
smartctl -a /dev/sda: https://ptpb.pw/E1ce

btrfs filesystem show

Label: none  uuid: 60ed664d-a9be-4576-a56f-6072defcc202
	Total devices 1 FS bytes used 69.87GiB
	devid    1 size 186.26GiB used 72.02GiB path /dev/sda4

btrfs device stats /

[/dev/sda4].write_io_errs    0
[/dev/sda4].read_io_errs     0
[/dev/sda4].flush_io_errs    0
[/dev/sda4].corruption_errs  0
[/dev/sda4].generation_errs  0

WorMzy · 2018-11-10 15:13:56

Okay, disk and filesystem look fine, so let's rule those out for now.

Some notes about your journal:

You haven't setup early microcode loading. This is highly recommended, although I can't say whether it will actually help in this case. See this.
Your default IO scheduler is cfq, but you have an SSD. Again this isn't necessarily a problem, but I would recommend using noop for the SSD. See this.

I see a lot of CPU overheating events, some lasting a few seconds, one lasting exactly a minute, do these times correspond at all to the behaviour you are seeing?

lis 09 00:06:25 archlinux kernel: CPU0: Core temperature above threshold, cpu clock throttled (total events = 1)
lis 09 00:06:36 archlinux kernel: CPU0: Core temperature/speed normal
lis 09 10:20:35 archlinux kernel: CPU0: Core temperature above threshold, cpu clock throttled (total events = 156)
lis 09 10:20:44 archlinux kernel: CPU0: Core temperature/speed normal
lis 09 12:09:44 archlinux kernel: CPU0: Core temperature above threshold, cpu clock throttled (total events = 646)
lis 09 12:09:47 archlinux kernel: CPU0: Core temperature/speed normal
lis 09 13:16:50 archlinux kernel: CPU0: Core temperature above threshold, cpu clock throttled (total events = 821)
lis 09 13:17:09 archlinux kernel: CPU0: Core temperature/speed normal
lis 10 10:12:29 archlinux kernel: CPU0: Core temperature above threshold, cpu clock throttled (total events = 861)
lis 10 10:13:29 archlinux kernel: CPU0: Core temperature/speed normal

Papi · 2018-11-10 17:00:37

* I have early microcode loading.
* I've changed I/O scheduler to noop. Thanks for the tip.
* Times of CPU overheating events doesn't correspond to my issue. I don't know why they happens, because my CPU temperature is between 40 and 70 °C

Papi · 2018-11-13 22:19:32

I did next experiments and here is dmesg output: https://ptpb.pw/fSLE.
There are a lot of messages like `INFO: task btrfs-transacti:247 blocked for more than 120 seconds.`

Papi · 2018-11-15 07:59:43

Solved...
I think it was issue in my btrfs filesystem. I've switched from btrfs to ext4 and now it's ok.

Possible fix for others: Create new btrfs filesystem and migrate your data with rsync or something like that.

Arch Linux

#1 2018-11-10 10:12:26

[SOLVED] Random I/O freezes

#2 2018-11-10 10:47:36

Re: [SOLVED] Random I/O freezes

#3 2018-11-10 11:57:57

Re: [SOLVED] Random I/O freezes

#4 2018-11-10 15:13:56

Re: [SOLVED] Random I/O freezes

#5 2018-11-10 17:00:37

Re: [SOLVED] Random I/O freezes

#6 2018-11-13 22:19:32

Re: [SOLVED] Random I/O freezes

#7 2018-11-15 07:59:43

Re: [SOLVED] Random I/O freezes

Board footer