[solved] Odd Out of Memory event, how do I track the cause

jwrichards82 · 2023-04-12 01:22:29

I have been going through a lot of forum posts, reading bits of the wiki and seeking help with the offending program.
The program is kinda new (at least to me) Raze, a Buiold Engine game sourceport based on ZDoom/GZDoom.
When I set it up and run it, my entire system locks up, requiring me to physically power cycle my computer OR just let it sit until the program stops. If my swap partition is turned off, it just happens faster. Running system monitor, I see the memory and CPU usage go way up, and using dmesg, I see a notification the process was killed by the Out of memory reaper.
Beyond that, I have no idea why this is happening. As far as I have noticed, it is the only program that does it, but it has happened before when my system completely locks up and I need to reboot. Whether it is the same issue or not, I cannot say at this time. One of the users at the ZDoom forums mentioned

I've had system freezes like that (not from Raze, but generally speaking) as a result of running out of memory (both physical and swap). Linux has a rather obscure bug where sometimes when memory gets exhausted, it'll enter this painfully slow loop of swapping things in and out of RAM while trying to run the OOM killer, during which time the system becomes unresponsive. It can take minutes (and in extreme circumstances, tens of minutes) for it to actually kill the offending process and let the system continue. It doesn't happen all the time, it seems to affect certain users more than others, and some never see the problem even when trying to trigger it.

Basic specs:
PU i5-660k (no overclock)
RAM 16GB DDR4-2400 CL14
GPU MSI GAMING X Radeon RX 580 8 GB

I am on KDE Plasma with systemd and my setup was mostly guided by Archfi. When I buy some new components and build a new computer, I plan on building an Arch setup without a guided installer so I might understand more and know 100% exactly what gets installed. If you need anymore information about my setup, please let me know what is needed.

What I am looking for: Is there some kind of script of logging program that I can run before I start Raze? I cannot find any documentation on Raze if there is some kind of debug mode or command variable that outputs a more detailed log. I would like to know if it is my setup with a latent bug or if it is an issue with the program. I don't think my search terms are getting what I need in Google as everything I read seems to not be relevant.

Last edited by jwrichards82 (2023-04-22 09:03:55)

seth · 2023-04-12 06:27:32

mostly guided by Archfi

https://bbs.archlinux.org/misc.php?action=rules

If you're trying to debug the game, valgrind is your friend.
Possible cause ****might**** be https://bbs.archlinux.org/viewtopic.php … 2#p2063182

Running system monitor

Running "top" and sorting by memory is maybe more useful…

One of the users at the ZDoom forums mentioned

https://wiki.archlinux.org/title/Improv … conditions

Is there some kind of script of logging program that I can run before I start Raze?

https://aur.archlinux.org/packages/collectd
https://archlinux.org/packages/?name=nmon
https://wiki.archlinux.org/title/List_o … m_monitors

jwrichards82 · 2023-04-20 07:02:59

seth wrote:

mostly guided by Archfi
https://bbs.archlinux.org/misc.php?action=rules

My apologies. I must have missed that. I will be rebuilding Linux soon without the script. I feel confident enough in Linux now, so...yeah. I will fix that.

As for the other things, I will keep it in mind. Once I am rebuilt following the initial steps outlined in the wiki and get to a state where I can test it; I'll re-post. Should I just label this one [closed] and repost a new topic, or just leave this one alone and update when I have more useful info? In talking to some people outside the forums, when it comes to Arch it is preferred to not use the various "easy install scripts" and such as it could still introduce some weird variable that very few users might see, and much worse with a distribution.

seth · 2023-04-20 07:32:14

You can keep the thread open and you can just try to disable zswap immediately, but if the game goes rogues and starts leaking RAM, there's likely a bug in the game and you'll have to valgrind it (or the game developer has)

jwrichards82 · 2023-04-21 02:24:02

Alright. That was easier done than I thought. Following just the Arch wiki, I got everything installed, also a new clean /home for my username, and I am still having issues with the program.

I decided to try and use Arch without SWAP after reading more, as I have enough memory for my uses and for it only being DDR4, CL14 is good and quite fast for its type, at least Cyberpunk 2077 in Proton is smooth as butter and I am not complaining being between 40 - 45 fps

I managed to get zswap turned off with kernel commands on boot, so it should be off right away.
However, because i made the mistake of having a swap partition, I can only get it to turn off by command in terminal once I am in Plasma. I followed the wiki, but either I am confused, an idiot or suffering frustration fatigue (I just spent an hour trying to figure out why my bluetooth headset would work; I can't get the system to stop turning on SWAP. I commented-out the swap l in fstab (in case i need it again), reboot and it is still active. Following https://wiki.archlinux.org/title/swap, I run

systemctl --type swap

And I get:

  UNIT                                   LOAD   ACTIVE SUB    DESCRIPTION   
  dev-disk-by\x2ddiskseq-1\x2dpart2.swap loaded active active Swap Partition

I then run

[james@arch042023 ~]$ systemctl disable \x2ddiskseq-1\x2dpart2.swap
Failed to disable unit: Unit file x2ddiskseq-1x2dpart2.swap does not exist.
[james@arch042023 ~]$ systemctl disable x2ddiskseq-1\x2dpart2.swap
Failed to disable unit: Unit file x2ddiskseq-1x2dpart2.swap does not exist.
[james@arch042023 ~]$ systemctl disable \x2dpart2.swap
Failed to disable unit: Unit file x2dpart2.swap does not exist.
[james@arch042023 ~]$ systemctl disable x2dpart2.swap
Failed to disable unit: Unit file x2dpart2.swap does not exist.
[james@arch042023 ~]$ systemctl disable x2dpart2
Failed to disable unit: Unit file x2dpart2.service does not exist.
[james@arch042023 ~]$ systemctl disable x2ddiskseq-1
Failed to disable unit: Unit file x2ddiskseq-1.service does not exist.

I then notice the article mentions to "mask" it.
So I do the same thing almost, swapping 'disable' with 'mask' with the response "Unit dev-disk-byx2ddiskseq-1x2dpart2.swap does not exist, proceeding anyway." (with varying degrees of the unit I THINK I am supposed to use.)

To lessen any possible issues I might cause, I have made sure to abort the process each time.

I tried using 'systemctl mask /dev/sda2', my logic thinking that is the corresponding unit name since that is the partition that was setup for SWAP. It let it go to /dev/null and thinking I did something right, I reboot and... SWAP is activated again.

What am I doing wrong or not understanding?
Again, maybe this is just fatigue or extreme frustration... Until I get swap completely turned off, I am not sure I can properly test the original problem yet

seth · 2023-04-21 07:05:41

I decided to try and use Arch without SWAP

That's stupid and not relevant to the subject of this thread.

I have enough memory for my uses

… and currently using zswap. Also the nv swap is used fo eg. hibernation.

smooth as butter

I hate butter. It's greasy and smells.
Disgusting.

I managed to get zswap turned off with kernel commands on boot, so it should be off right away.

Good. You can stop there.

What am I doing wrong or not understanding?

That the theorized problem is only related to zswap.

Fyi, "dev-disk-by\x2ddiskseq-1\x2dpart2.swap" has tokens ("\") that act as escape sequences in systemd AND bash (and zsh and every other shell you likely use)
So you'll have to quote it

systemctl mask 'dev-disk-by\x2ddiskseq-1\x2dpart2.swap'

But again: not relevant at all.

jwrichards82 · 2023-04-21 08:22:38

And this is why I like your responses to posts. You don't beat around the bush.

I also considered turning off some of the swap functions as I will never use hibernation. BUT. I see the wisdom. I am going to be getting a laptop, I never had one outside of the jobs I have had, and it will also be running Arch. I may want to use it on mobile devices, so I might as well be in the habit.

I will stop what I am doing with the swap then. I already ruled out swap partition causing issues as the only change in my tests with raze is OOM kicks in faster when Swap partition is off.

I think I might be setting this as closed then. I am waiting on the dev and a couple other users who have Arch installs to run their tests on the ZDoom forums.

Thank you for your time! It was a pleasure learning the last few days. I will wait for a couple more messages form them and I will come back here to close or continue.

seth · 2023-04-21 11:53:32

run their tests on the ZDoom forums

Is this acutally zdoom? Afaiu "raze" is some g/zdoom fork?

Make sure the RAM consuming offender is actually the game directly (and not eg. Xorg or the kernel) and try its behavior w/ its default settings for reproducibility.

jwrichards82 · 2023-04-21 15:09:15

Yes. Raze is base don GZDoom tech to play Build Engine games. Main GZDoom runs just fine. I have even tried breaking things, turning on tons of features and high-res textures. My computer slows down, doesn't lock up due to aging hardware.

I am going to install both the Zen and LTS kernels and try each. One of my friends suggested to switch to Wayland as it should be matured enough.

In all my testing, it is with default settings and instructions on getting assets from the various games. It isn't a per game bug as one person suggested, I get the same behavior with Duke Nukem 3D, Blood, Shadow Warrior and Redneck Rampage assets. One of the forum goers also says they use PopOS and has not had a single problem, so I am also waiting on what libraries at what version they are using.

UPDATE

Well, I was able to go through all the dev builds, daily if not multiple time sin the day when they would make builds from git when they make a change. Turns out, something changed 13 days ago. I can play on older versions, but after that date, the program "breaks"
So. I will be setting this tread as solved.

Thank you much for your time, Seth!

Last edited by jwrichards82 (2023-04-21 18:02:47)

2ManyDogs · 2023-04-21 22:08:53

Please do not mark a thread [closed] -- that is used by staff when a thread is not open for further replies. If your problem is solved, please add [SOLVED] to the title.

Arch Linux

#1 2023-04-12 01:22:29

[solved] Odd Out of Memory event, how do I track the cause

#2 2023-04-12 06:27:32

Re: [solved] Odd Out of Memory event, how do I track the cause

#3 2023-04-20 07:02:59

Re: [solved] Odd Out of Memory event, how do I track the cause

#4 2023-04-20 07:32:14

Re: [solved] Odd Out of Memory event, how do I track the cause

#5 2023-04-21 02:24:02

Re: [solved] Odd Out of Memory event, how do I track the cause

#6 2023-04-21 07:05:41

Re: [solved] Odd Out of Memory event, how do I track the cause

#7 2023-04-21 08:22:38

Re: [solved] Odd Out of Memory event, how do I track the cause

#8 2023-04-21 11:53:32

Re: [solved] Odd Out of Memory event, how do I track the cause

#9 2023-04-21 15:09:15

Re: [solved] Odd Out of Memory event, how do I track the cause

#10 2023-04-21 22:08:53

Re: [solved] Odd Out of Memory event, how do I track the cause

Board footer