[SOLVED] Why does my system let GHC almost taking it down?

Enrico1989 · 2026-06-23 13:54:28

This is the output of lscpu

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           43 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  32
On-line CPU(s) list:                     0-31
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen Threadripper 1950X 16-Core Processor
CPU family:                              23
Model:                                   1
Thread(s) per core:                      2
Core(s) per socket:                      16
Socket(s):                               1
Stepping:                                1
Microcode version:                       0x8001129
Frequency boost:                         enabled
CPU(s) scaling MHz:                      66%
CPU max MHz:                             3400.0000
CPU min MHz:                             2200.0000
BogoMIPS:                                6786.53
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sev
L1d cache:                               512 KiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                8 MiB (16 instances)
L3 cache:                                32 MiB (4 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-7,16-23
NUMA node1 CPU(s):                       8-15,24-31
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT vulnerable
Vulnerability Spec rstack overflow:      Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Vulnerable

and this is the output of lsmem

RANGE                                  SIZE  STATE REMOVABLE  BLOCK
0x0000000000000000-0x000000007fffffff    2G online       yes   0-15
0x0000000100000000-0x000000087fffffff   30G online       yes 32-271

Memory block size:                128M
Total online memory:               32G
Total offline memory:               0B
Memmap on memory parameter:         no

I'm working on a Haskell project, and when I build it, I go for

cabal build all -jN

As regards N, if I use 32 because my system has 32 CPUs, the system becomes fairly unresponsive, and some other applications are negatively impacted not just in terms of responsiveness, but also in that pieces of them start "crashing".

Here a report of some browser's tabs being SIGKILLed. It turned out it's the OOM killer doing its job.

But my question is: why is my system allowing this?

Yes, I'm the one passing -j32, but my intention is more to tell the system "don't limit yourself to using less than 32 cores", but without encouraging it to steal such cores away from other applications.

Am I just misunderstanding what's expected of a program when telling it to use all cores? Or is it that GHC is too aggressively stealing computational resources? Or something else?

Last edited by Enrico1989 (2026-06-29 11:07:56)

seth · 2026-06-23 14:17:05

But my question is: why is my system allowing this?

Because it doesn't know what you want.
https://wiki.archlinux.org/title/Cgroups
https://wiki.archlinux.org/title/Improv … conditions
https://www.baeldung.com/linux/memory-o … oom-killer

Raising the (io)nice'ness of the process will make it not/less steal CPU time but still use everything it's afforded.
nb. that a bigger problem w/ the massive parallel execution will be the RAM limitation - if you're running OOM the kernel has to kill stuff to free memory for the other consumers, but if you constrain the RAM access of a process (group) and glibc can no longer claim the demanded memory, there's a tremendous chance the process will just crash/abort.

marcoe · 2026-06-23 15:34:53

Try running GHC in its own cgroup:

systemd-run --scope --slice-inherit cabal build all -jN ...

If that doesn't appear to help, check if it's memory bandwidth related:

for i in {1..8}; do memhog -r40 200M & done

And slowly increase the number of memhog workers, for example my system barely holds 13 workers - the responsiveness of everything drops dramatically after that - I can also observe it with ffmpeg encoding, it's just not worth it to spawn more than 16 ffmpeg encoding threads, because they are bottlenecked by memory bandwidth and as far as I know there is no (software) remedy for that -- are you sure using all 32 threads helps with compile times?

Last edited by marcoe (2026-06-23 15:41:23)

Lone_Wolf · 2026-06-23 18:07:49

Thread(s) per core:                      2
Core(s) per socket:                      16

Total online memory:               32G

I had a 1920x threadripper with 12/24 c/t and 16 Gib .
Had to add swap of 24 Gib and reduce number of cores used in compiling to 18 to avoid OOM crashes.

Currently am using a ryzen 9 9950X with 16/32 C/T and 64 GiB .
No need to reduce number of cores used anymore .

Comments online related to 'ninja being very greedy and bringing system down' suggests each job takes 1 to 2 Gib of memory .
automake is / used to be better at managing lots of jobs but most projects have switched to cmake or meson .

How much swapspace does your system have ?
If it's less then 16 GiB increase it to that and test. (Don't be afraid to increase to 32 GiB if it still crashes) .

Enrico1989 · 2026-06-24 13:14:28

Lone_Wolf wrote:

How much swapspace does your system have ?
If it's less then 16 GiB increase it to that and test. (Don't be afraid to increase to 32 GiB if it still crashes) .

$ swapon --show
NAME      TYPE       SIZE USED PRIO
/dev/sda5 partition 15.6G 1.3G   -1
 
$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi        17Gi       5.6Gi       201Mi       8.6Gi        13Gi

However,

sudo fdisk -l /dev/sda 
Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 850 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 4F756C60-7752-47C8-ADD4-1554015CF6BA

Device         Start       End   Sectors   Size Type
/dev/sda1       2048   1023999   1021952   499M Windows recovery environment
/dev/sda2    1024000   1228799    204800   100M EFI System
/dev/sda3    1228800   1261567     32768    16M Microsoft reserved
/dev/sda4    1261568 464773119 463511552   221G Microsoft basic data
/dev/sda5  464773120 497541119  32768000  15.6G Linux swap
/dev/sda6  497541120 871915519 374374400 178.5G Linux filesystem
/dev/sda7  871915520 976773119 104857600    50G Linux filesystem

so with the swap in between there... can I even give it more space without risking screwing everything up? Sounds risky...

Last edited by Enrico1989 (2026-06-24 13:15:13)

Lone_Wolf · 2026-06-24 13:37:36

If one of your linux partitions has enough room you can create a swapfile .
Linux swap system is smart enough to combine multiple swapareas. Lower on that same page is detailed how you can instruct linux which swaparea to use first.

Other workarounds :
reduce the memory used for temporary files by directing TMPDIR to a physical location .

disable LTO and/or disable debug builds

seth · 2026-06-24 13:54:56

I'm not sure whether the build will speed up if the price for like 1/4 more jobs is swap access - even nvme's are slow compared to RAM (iirc the fastestmostexpensive ones are 1/5 of DDR4 or so…)

Lone_Wolf · 2026-06-24 14:16:13

Finishing the job is the important thing in my opinion .

For some progams I had to use big swapspace + tmpdir on physical + disable lto + disable debug + reduce number of cores used .

The 64 GiB memory on my current desktop was supposed to be 128 GiB and cost less than a 1000 euros, but it ended up to cost 1200 for half the amount (bought in decmber last year) .
Still I'm very glad I didn't fall back to 48 or 32 GiB .

seth · 2026-06-24 14:28:57

Yeah but the OP is deliberately runnin -j32 and afaiu doesn't run into problems w/ lower job counts.
He seems to run OOM, thrashing and eventually the OOM killer, but since the first complaint is that "the system becomes fairly unresponsive" that's not going away if you systematically run on swap.
=> the build might overall be more performant by lowering the job count and keeping the RAM demand at bay to avoid the extremely costly swap.

Enrico1989 · 2026-06-24 17:43:37

seth wrote:

Yeah but the OP is deliberately runnin -j32

Yep, for the reason I explained.

seth wrote:

and afaiu doesn't run into problems w/ lower job counts.

Well, surely with -j16 the system remains fairly responsive, but I wouldn't say I don't notice any difference. Maybe with -j8 I don't.

I should add that the unresponsiveness doesn't spike immediately when I run the -j32 build command, but after a bunch of seconds. It feels like some specific, but not short, part of the build is being particularly critical. Is there anythink I can dump to file while the system is under stress that could help understanding?

seth wrote:

keeping the RAM demand at bay to avoid the extremely costly swap.

Should I check how much swap is used when the system becomes unresponsive? I mean, if it becomes unresponsive, checking is not easy, though.

seth · 2026-06-24 19:37:01

Yes, I'm the one passing -j32, but my intention is more to tell the system "don't limit yourself to using less than 32 cores", but without encouraging it to steal such cores away from other applications.

Run the compilation w/ maximum (io)nice'ness and track RAM and swap usage.
If the unresponsiveness was driven by the CPU or IO you can expect that problem to disappear.

If it's driven by RAM/SWAP the answer is that you need either more RAM or less parallel jobs.
You can memory quota the jobs and leave them w/ unlimited swap and you'd have to run tests on the actual process, but the rule of thumb is that once you start swapping everything grinds to a halt and having more cores wait for malloc won't make anything faster.

Enrico1989 · 2026-06-24 20:12:48

seth wrote:

Run the compilation w/ maximum (io)nice'ness and track RAM and swap usage.

Than means running

ionice -c 3 the-compilation-command

right?

seth · 2026-06-24 20:18:42

https://man.archlinux.org/man/nice.1 and https://man.archlinux.org/man/ionice.1 are different things and you want both (and to be as nice as possible)
"ionice nice whatever" will get the job done (using the idle ionice might severely stall the build)

Enrico1989 · 2026-06-25 08:10:34

seth wrote:

Run the compilation w/ maximum (io)nice'ness and track RAM and swap usage.
If the unresponsiveness was driven by the CPU or IO you can expect that problem to disappear.

I've run

ionice nice cabal build all -j32

and I had the impression that things were better, but maybe I just hadn't observed htop during the initial phase, before. What I've seen is that, at least initially, all CPU usage bars had a lot of blue, which according to htop's help means "low". Then at some point the system became unresponsive for a few minutes, until the build completed, at which point it recovered, but some browser tabs had crashed in the meanwhile.

Anyway, maybe I should ask what you mean by "track RAM and swap usage" other than having a look at htop.

seth wrote:

If it's driven by RAM/SWAP the answer is that you need either more RAM or less parallel jobs.

I suppose I have to accept this

seth wrote:

You can memory quota the jobs and leave them w/ unlimited swap

"Memory quota" as a verb means "limit the RAM they can use", and "leave them w/ unlimited swap" means they use the swap for whatever additional "memory" they need, right? It doesn't sound like a great solution, as you also suggest.

seth · 2026-06-25 10:11:06

Anyway, maybe I should ask what you mean by "track RAM and swap usage" other than having a look at htop.

Not much - in this context a broad oversight over the development as provided by those aggregators is sufficient.

"limit the RAM they can use", and "leave them w/ unlimited swap" means they use the swap for whatever additional "memory" they need, right?

Right.
You're protecting the rest of the system by restricting the resources of the build job (and redirect the inevitable sluggishness there)

Enrico1989 · 2026-06-29 11:07:37

Enrico1989 wrote:

I've run
ionice nice cabal build all -j32
and I had the impression that things were better, but maybe I just hadn't observed htop during the initial phase, before. What I've seen is that, at least initially, all CPU usage bars had a lot of blue, which according to htop's help means "low". Then at some point the system became unresponsive for a few minutes, until the build completed, at which point it recovered, but some browser tabs had crashed in the meanwhile.

It might be contingent, but since I wrote this message, thing seem to go better. I'll marked this as SOLVED for now.

Arch Linux

#1 2026-06-23 13:54:28

[SOLVED] Why does my system let GHC almost taking it down?

#2 2026-06-23 14:17:05

Re: [SOLVED] Why does my system let GHC almost taking it down?

#3 2026-06-23 15:34:53

Re: [SOLVED] Why does my system let GHC almost taking it down?

#4 2026-06-23 18:07:49

Re: [SOLVED] Why does my system let GHC almost taking it down?

#5 2026-06-24 13:14:28

Re: [SOLVED] Why does my system let GHC almost taking it down?

#6 2026-06-24 13:37:36

Re: [SOLVED] Why does my system let GHC almost taking it down?

#7 2026-06-24 13:54:56

Re: [SOLVED] Why does my system let GHC almost taking it down?

#8 2026-06-24 14:16:13

Re: [SOLVED] Why does my system let GHC almost taking it down?

#9 2026-06-24 14:28:57

Re: [SOLVED] Why does my system let GHC almost taking it down?

#10 2026-06-24 17:43:37

Re: [SOLVED] Why does my system let GHC almost taking it down?

#11 2026-06-24 19:37:01

Re: [SOLVED] Why does my system let GHC almost taking it down?

#12 2026-06-24 20:12:48

Re: [SOLVED] Why does my system let GHC almost taking it down?

#13 2026-06-24 20:18:42

Re: [SOLVED] Why does my system let GHC almost taking it down?

#14 2026-06-25 08:10:34

Re: [SOLVED] Why does my system let GHC almost taking it down?

#15 2026-06-25 10:11:06

Re: [SOLVED] Why does my system let GHC almost taking it down?

#16 2026-06-29 11:07:37

Re: [SOLVED] Why does my system let GHC almost taking it down?

Board footer