[SOLVED] System Freeze when running rsync with kernel-ck

mclang · 2016-09-10 17:49:46

Couple days ago I tried to run my usual backup script that uses rsync.

I was really suprised when I later found that my whole computer had freezed totally. I tried several times again and every time the result was same - I had to reboot my computer using power button. Of course rsync managed to copy a little bit more every time, but the whole process never finished. I also tried if boinc, steam games or movies would trigger the same behaviour but in vain.

I tried to find reason with journalctl but the last thing there was me sudoing to root. Then today I remembered that I have also the stock kernel installed, so why not try with that... And everything worked perfectly.

CK kernel packages:

linux-ck-piledriver 4.7.3-3
nvidia-ck-piledriver 370.28-1

Stock kernel packages:

linux 4.7.2-1
nvidia 370.28-1

Does anybody know what the reason could be and how to fix this?

Last edited by mclang (2016-09-15 06:53:25)

JohnBobSmith · 2016-09-11 02:00:24

Your two kernels are the not the same. The linux-ck kernel patch set itself should have no impact on rsync as far as I'm aware. However, the newer (newest?) kernel 4.7.3-3 is likely to blame. The solution is to use an older kernel, and file bug report(s) accordingly.

mclang · 2016-09-11 12:32:35

Good point.

But couldn't it be BFS 480 related? Thinking this because in the patch notes Con says that this is biggest update in BFS in long time... Can I test this somehow?

I almost got confused between BFS and BFQ but luckily read Linux-CK page of Arch Wiki again... My SSD is using noop, but the two HDD drives that are used when rsynching use bfq, so maybe I should also test with other IO scheduler?

JohnBobSmith · 2016-09-11 14:22:39

Hm... The most important thing for testing is that you are using the same kernel when changing schedulers and related programs. Therefore, I would test with the stock arch kernel and add/change things until something breaks. Then, compare the point at which the arch kernel broke to the linux-ck kernel. Looks for packages, configuration changes, version changes, what have you. For example, if BFS version XYZ breaks rsync on the stock kernel and the linux-ck kernel uses BFS version XYZ, you can be pretty sure thats the source of the problem. I hope that makes sense. Good luck with the troubleshooting

Last edited by JohnBobSmith (2016-09-11 14:23:46)

graysky · 2016-09-11 14:56:01

@OP - Please try linux-ck-4.7.3-4 (AUR now, repo-ck in several hours).

mclang · 2016-09-11 16:58:38

I don't know how to use BFS and BFQ with stock kernel.

I do know how to use other I/O schedulers than BFQ while using linux-ck, but I will first try linux-ck-4.7.3-4 when it becomes available in repo-ck and report back.

Last edited by mclang (2016-09-11 17:44:54)

graysky · 2016-09-11 18:02:14

Online now.

Last edited by graysky (2016-09-11 18:14:01)

mclang · 2016-09-12 18:11:02

Sadly updating to linux-ck-4.7.3-4 didn't help.

Backing up 22G folder with rsync still freezes computer after a while. I will try different I/O schedulers tomorrow if I have time. Is there way to switch easily to stock kernel scheduler instead of BFS so that I could try if BFS is the problem?

And by the way, the drives I use are both formatted as XFS, it that makes any difference.

graysky · 2016-09-12 19:11:53

CPU schedulers: BFS and CFS. They are not tuneable.
I/O schedulers: BFQ, CFQ, noop, and deadline. They are tuneable. The linux-ck package ships with support for BFQ but you have to enable it.

Which IO scheduler are you using? I recommend that you preform your tests with CFQ since it is the kernel default. To test, you can boot into the stock kernel and try your 22G rsync. Then boot into the linux-ck kernel and repeat. Again, if you aren't using BFQ, you may already have the linux-ck data point.

halocaridina · 2016-09-13 01:53:21

Sorry for the "me too", but I've also experienced several freezes over the last few days when booted into linux-ck-silvermont 4.7.3-[2-4] from repo-ck.

First instances happened with 4.7.3-2 while rsync'ing to clone the system (according to https://wiki.archlinux.org/index.php/Fu … with_rsync) from a smaller to larger SSD. I initially thought that the USB 3.0 port was autosuspending, thus killing the transfer and freezing the system in the process. Following two consecutive freezes while the rsync was running unattended, I sat and randomly moved the cursor on the screen for the next two (i.e., to complete transfer of / then /home) rsync operations (again thinking that USB autosuspend might be the reason). These last two rsync operations completed without the system freezing.

Next feeze happened suddenly the following day under 4.7.3-3 while in a terminal on the newly cloned system. Typing one moment and system frozen the next. This prompted a look at the AUR linux-ck page and finding that others were also experiencing random freezes. This was around the time 4.7.3-4 was released (late evening of 9/11/16), so upgraded and shutdown the laptop for the evening. FYI on the hardware/system as of this posting:

System:    Host: wopr Kernel: 4.7.2-1-ARCH x86_64 (64 bit gcc: 6.1.1) Desktop: i3 4.12 Distro: Arch Linux
Machine:   System: Dell (portable) product: Latitude 3150
           Mobo: Dell model: 05R1X2 v: A00 UEFI: Dell v: A06 date: 08/29/2016
Battery    BAT0: charge: 29.3 Wh 86.3% condition: 34.0/34.0 Wh (100%) model: Samsung SDI DELL R5MD0D9 status: N/A
CPU:       Quad core Intel Pentium N3540 (-MCP-) cache: 1024 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 17314
           clock speeds: max: 2665 MHz 1: 499 MHz 2: 499 MHz 3: 499 MHz 4: 499 MHz
Graphics:  Card: Intel Atom Processor Z36xxx/Z37xxx Series Graphics & Display bus-ID: 00:02.0
           Display Server: X.Org 1.18.4 driver: N/A Resolution: 1366x768@60.06hz
           GLX Renderer: Mesa DRI Intel Bay Trail GLX Version: 3.0 Mesa 12.0.2 Direct Rendering: Yes
Audio:     Card Intel Atom Processor Z36xxx/Z37xxx Series High Definition Audio Controller
           driver: snd_hda_intel bus-ID: 00:1b.0
           Sound: Advanced Linux Sound Architecture v: k4.7.2-1-ARCH
Network:   Card-1: Intel Wireless 7265 driver: iwlwifi bus-ID: 03:00.0
           IF: wlp3s0 state: up mac: <filter>
           Card-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           driver: r8169 v: 2.3LK-NAPI port: e000 bus-ID: 04:00.0
           IF: enp4s0 state: down mac: <filter>
Drives:    HDD Total Size: 525.1GB (15.8% used) ID-1: /dev/sda model: Crucial_CT525MX3 size: 525.1GB temp: 37C
Partition: ID-1: / size: 30G used: 9.3G (34%) fs: ext4 dev: /dev/dm-1
           ID-2: /boot size: 511M used: 109M (22%) fs: vfat dev: /dev/sda1
           ID-3: /home size: 452G used: 69G (16%) fs: ext4 dev: /dev/dm-2
Sensors:   System Temperatures: cpu: 46.0C mobo: N/A
           Fan Speeds (in rpm): cpu: N/A
Info:      Processes: 163 Uptime: 5:03 Memory: 561.3/7875.1MB Init: systemd Gcc sys: 6.2.1
           Client: Shell (zsh 5.2) inxi: 2.3.1

As shown in the above info, I'm currently on the stock Arch kernel following a freeze this afternoon while booted into 4.7.3-4 (again happened while in a terminal with nothing CPU or I/O intensive running, just a sudden hard freeze).

In all instances, absolutely nothing in the journal suggesting the underlying cause. I wanted to chime in since there is some discussion in the "Unofficial linux-ck repo" thread:

https://bbs.archlinux.org/viewtopic.php?id=111715

that these freezes might be specific to haswell desktops. Doesn't appear so since I'm on a silvermont laptop. i'll keep an eye on these threads and move back to linux-ck once the dust settles.

Graysky, thanks for all the work on this package. Its greatly appreciated.

Halocaridina

agm28011997 · 2016-09-13 09:02:20

I am in linux ck 4.7.3-4 now, the last time I used I got 3 freezes but I am testing again for ensure me that the kernel is bringing me the problems and not other things like drivers(that are in the mainline too) or programs, (I play in ck and in the mainline) at that moment i have powered on from 20 minutes ago, any freeze yet.. I saw the two topics I have the same reproducible things that the person have in the arch oficial kernel but I have not any freeze in the oficial.. at least I had this luck, I had the freezes only in ck since the bfs 480 patch, and other thing, gravysky and con kolivas, thanks for your work in the patchs and the packages, i am testing all the thing I can for view where are the problems, I have put a virtual machine without freezes, I don't know if it was luck but I willl try again, I will help you as much as I can

agm28011997 · 2016-09-13 09:06:31

other thing is that at the moment any freeze i will see if there are freezes only when I am gaming or in general

agm28011997 · 2016-09-13 10:01:52

well, I can tell now that the freezes occur with normal work apart from gaming, included the last kernel, and for me with the oficial kernel and the older of ck with the same git drivers and mesa git all is stable..

mclang · 2016-09-13 11:36:43

Stock kernel with CFQ was stable.

I'm using BFQ with my two HDDs and noop with my SDD. HDDs have XFS filesystem and SDD, whoich holds my root filesystem, is btrfs. Processor is AMD FX6300, so the kernel I use is linux-ck-piledriver 4.7.3-4. I haven't experienced system freeze during any other activity than rsynching large directories.

I will test rsync with stock kernel and linux-ck with CFQ tonight if I have time. I try to manage also test some gaming, e.g with new System Shock reboot demo.

If there is any other info I can provide, let me know.

Forgot to add that when the system freezes REISUB keysequence does not work

Last edited by mclang (2016-09-13 11:46:56)

mclang · 2016-09-14 04:03:22

It seems that BFQ is not the problem.

I changed BFQ -> CFQ for my HDD drives and made a loop that deletes the backup and does it again every five minutes. After three loops computer froze again. I didn't have time to play any games, but I leave a game running for the day and report back when I get home from work.

Is there any other threads discussing this that I should be aware of?

And now I noticed that maybe I should go http://ck-hack.blogspot.com/2016/09/bfs … 7-ck2.html with this... I also found other threads about freezes also with stock kernel.

Last edited by mclang (2016-09-14 14:21:13)

mclang · 2016-09-15 06:52:03

Bug fixed I think.

I have been running rsync backup tests with linux-ck-piledriver 4.7.3-5 installed from Graysky's "linux-ck" repository and I haven't encountered any freezes. I seems thus that ck4/bfs497 finally fixed things, for me at least.

Arch Linux

#1 2016-09-10 17:49:46

[SOLVED] System Freeze when running rsync with kernel-ck

#2 2016-09-11 02:00:24

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#3 2016-09-11 12:32:35

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#4 2016-09-11 14:22:39

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#5 2016-09-11 14:56:01

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#6 2016-09-11 16:58:38

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#7 2016-09-11 18:02:14

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#8 2016-09-12 18:11:02

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#9 2016-09-12 19:11:53

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#10 2016-09-13 01:53:21

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#11 2016-09-13 09:02:20

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#12 2016-09-13 09:06:31

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#13 2016-09-13 10:01:52

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#14 2016-09-13 11:36:43

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#15 2016-09-14 04:03:22

Re: [SOLVED] System Freeze when running rsync with kernel-ck

#16 2016-09-15 06:52:03

Re: [SOLVED] System Freeze when running rsync with kernel-ck

Board footer