Total system freeze caused by application

andrzej1_1 · 2019-03-20 22:18:12

Hello, since few days I am trying to run Unreal Engine editor, but system freezes completely every time I open project. Obviously I tested common problems:
- heavy CPU usage -> nope, I run application with lowest priority "nice -n19"; I also tried cgroup limit
- heavy IO -> nope, LED does not indicate any operations; I also tried "ionice -c 3"
- low memory -> nope, I have 32 GB of RAM and restricting it with cgroups doesn't change a thing

As usual I tried to handle freeze with SysRq keys, but they doesn't work. Causing such freeze shouldn't be possible from userspace, so I assume it's kernel issue.

What I have already tried:
- changing kernel 5.0.2 to LTS version (4.19.30)
- switching i3 to KDE
- using different hardwares:
a) desktop -> CPU: i7-8700, GPU: integrated, RAM: 32 GB
b) laptop -> CPU: i5-8250u, GPU: Radeon 520, RAM: 8 GB
- disabling dkms modules

I decided to enable insane kernel debugging:

 ### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux'
    ...
linux   /vmlinuz-linux root=UUID=[REDACTED] ro rootwait ignore_loglevel debug debug_locks_verbose=1 sched_debug initcall_debug mminit_loglevel=4 udev.log_priority=8 loglevel=8 earlyprintk=vga,keep log_buf_len=10M print_fatal_signals=1 apm.debug=Y i8042.debug=Y drm.debug=1 scsi_logging_level=1 usbserial.debug=Y option.debug=Y pl2303.debug=Y firewire_ohci.debug=1 hid.debug=1 pci_hotplug.debug=Y pci_hotplug.debug_acpi=Y shpchp.shpchp_debug=Y apic=debug show_lapic=all hpet=verbose lmb=debug pause_on_oops=5 panic=10

There are plenty of logs, but theys stops after boot is completed and Xorg started. So triggering freeze does not seem to give more information, but maybe I am using "journactl" incorrectly?

Could you give me any advice, what should I do to figure out problem?

user71383 · 2019-03-21 02:07:35

I'm having the same/similar problem since I installed ArchLinux with KDE Plasmashell (and ZFS) about 2 years ago.

Since the beginning I have problems with freezes (while importing databases to MariaDB). No logs and no indication of malfunction, just a total system freeze.

While using computation intensive applications I can go around the problem by killing the plasmashell process(es) with "killall plasmashell" and later restart the GUI with "kstart5 plasmashell"

I assume the problem connected with the swapping of memory (with ZFS) in connection with plasmashell, because I noticed a huge increase of used memory and hence a lot of swapping before every freeze.

Pls let me know if that helps …

Last edited by user71383 (2019-03-21 02:14:25)

Head_on_a_Stick · 2019-03-21 10:40:14

andrzej1_1 wrote:

I tried to handle freeze with SysRq keys, but they doesn't work

ArchWiki wrote:

To use these, they must first be activated

https://wiki.archlinux.org/index.php/Ke … uts#Kernel

andrzej1_1 wrote:

There are plenty of logs, but theys stops after boot is completed and Xorg started. So triggering freeze does not seem to give more information, but maybe I am using "journactl" incorrectly?

We have no way of knowing that unless you explain exactly how you are attempting to use the journal. Have you read the relevant ArchWiki page?

FWIW, my journal continues to record entries even after X is started (but I'm using dwm and Debian buster rather than Plasma and Arch Linux).

andrzej1_1 · 2019-03-21 14:04:17

@user71383 As I already said it's not low memory issue - I restricted memory usage with cgroups, so there is no way it can freeze kernel or X11.

@Head_on_a_Stick I was using SysRq many times and they are enabled with kernel parameter. Another reason indicating it's kernel problem is that capslock key is not handled when system is freezed - I mean that pressing key doesn't toggle keyboard LED.
You are right, I was using jorunalctl incorrectly. I enabled insane debug again and

$ sudo journactl -f

is printing all kernel logs as it supposed to.

I recorded video of freeze with logs enabled and I repeated it few times - it always stops at some DRM log.

seth · 2019-03-21 14:57:32

Did you update to mesa 19 "a few days" ago?
Did this actually ever work or did you just develop an interest in the unreal editor a few days ago?

andrzej1_1 · 2019-03-21 15:27:18

@seth I have never used it before. UE4 editor is making my system hung since first installation, which was on 18.03.2019. I updated mesa packages from 18.3.4-1 to 19.0.0-1 two days later (20.03.2019).

seth · 2019-03-21 15:43:41

The weird part is that an intel and a radeon chip are affected.
Do you use the modesetting ddx driver on both or xf86-video-{intel,amdgpu,ati}?

andrzej1_1 · 2019-03-21 16:15:20

$ pacman -Q | grep xf86-
xf86-input-libinput 0.28.2-1
xf86-video-ati 1:19.0.0-2
xf86-video-intel 1:2.99.917+863+g6afed33b-1
xf86-video-vesa 2.4.0-2

I don't have any config files inside /etc/X11/xorg.conf.d/ nor forced modesetting.
I uninstalled these packages, rebooted and freeze is still happening.

I think neither printing logs in console nor reading them from files is reliable, so I will try to use NetConsole.

Last edited by andrzej1_1 (2019-03-22 15:30:16)

seth · 2019-03-22 18:54:22

You could also try to strace the editor and see whether there's a pattern in the last system calls.
Do you use a raid or lvm on either system?

andrzej1_1 · 2019-03-23 19:23:48

@seth strace shows a lot of FUTEX_* and DRM_IOCTL_I915_GEM_* operations at the end, so nothing really useful. I don't use RAID nor LVM.

Fortunately with NetConsole I was able to capture GPU error:

...
[  387.161861] [drm] GPU HANG: ecode 9:0:0x8ed9fff2, in UE4Editor [3127], reason: hang on rcs0, action: reset
[  387.161866] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[  387.161868] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[  387.161870] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
[  387.161871] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
[  387.161873] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[  387.162881] i915 0000:00:02.0: Resetting rcs0 for hang on rcs0
...
[  387.164405] i915 0000:00:02.0: Resetting chip for hang on rcs0

I just reported issue and in the meantime I will look for a workaround.

seth · 2019-03-23 20:12:04

https://wiki.archlinux.org/index.php/In … RI3_issues ?

andrzej1_1 · 2019-03-23 21:19:30

Unreal Engine requires DRI3, otherwise it crashes.

seth · 2019-03-23 21:34:19

It's weird enough that you got a GPU hang on intel AND radeon…
Do you have access to some nvidia GPU?

andrzej1_1 · 2019-03-23 22:42:24

I don't have GeForce and no one around me don't have such card too - everyone is using laptops. I suppose there was something wrong with settings/drivers and UE4 used integrated GPU instead of ATI card. I need some time to investigate it.

Edit:
By default UE4 on Linux uses Vulkan, so editor was using Intel GPU, because it works without further configuration. That's why it's freezing both PC and laptop - both use mesa drive for Intel. To make Vulkan work with Radeon I had to switch from ATI to AMDGPU driver and remove vulkan-intel. Now application works without any problem on my laptop.

I doubt they will fix driver soon, because there are similar issues reported and not fixed since two years. However I found there is command line argument (-opengl) that forces UE4 to use OpenGL, which works with minor glitches.

Last edited by andrzej1_1 (2019-03-26 22:30:13)

Arch Linux

#1 2019-03-20 22:18:12

Total system freeze caused by application

#2 2019-03-21 02:07:35

Re: Total system freeze caused by application

#3 2019-03-21 10:40:14

Re: Total system freeze caused by application

#4 2019-03-21 14:04:17

Re: Total system freeze caused by application

#5 2019-03-21 14:57:32

Re: Total system freeze caused by application

#6 2019-03-21 15:27:18

Re: Total system freeze caused by application

#7 2019-03-21 15:43:41

Re: Total system freeze caused by application

#8 2019-03-21 16:15:20

Re: Total system freeze caused by application

#9 2019-03-22 18:54:22

Re: Total system freeze caused by application

#10 2019-03-23 19:23:48

Re: Total system freeze caused by application

#11 2019-03-23 20:12:04

Re: Total system freeze caused by application

#12 2019-03-23 21:19:30

Re: Total system freeze caused by application

#13 2019-03-23 21:34:19

Re: Total system freeze caused by application

#14 2019-03-23 22:42:24

Re: Total system freeze caused by application

Board footer