You are not logged in.
Trying to look for answers here on the Arch forums, since this affects me when building packages for projects which use cmake... I also want to note that I don't have much experience (or any at all TBH) with anything C/C++ related, so please bear with me.
I've been maintaining my own kwin and plasma-workspace PKGBUILDs for a long time now, since I'm applying a couple of patches for specific stuff which I don't agree with upstream. This is irrelevant to the problem though.
The problem is that when building kwin or plasma-workspace, my system always becomes super unresponsive and basically unusable. Since I don't have any problems when for example building the Linux kernel (also using my own PKGBUILD), I guess that the problem comes from cmake-based projects, like kwin and plasma-workspace.
My system's CPU is a Ryzen 3950X (16C32T), hence why I have set `MAKEFLAGS=-j32` in my `makepkg.conf`. As described, this works as expected when building the kernel for example, without any responsiveness issues while building. Process scheduling works perfectly fine and the system stays usable despite 100% CPU utilization.
However, building those cmake-based projects seems to ignore the number of parallel build-jobs. Setting the `-j32` parameter on the `cmake --build` command itself (or setting the `CMAKE_BUILD_PARALLEL_LEVEL=32` env var) also doesn't help. And surprisingly, reducing it to something lower like for example 8 also doesn't help (including `MAKEFLAGS=-j8` set at the same time). It always results in 100% CPU utilization, so it looks like (way) more than 32 jobs are run in parallel, choking the CPU scheduler and making the system totally unresponsive.
Setting `-j1` on `cmake --build` does not do that, but that's obviously not the solution. According to the docs/manual: "Some native build tools always build in parallel. The use of <jobs> value of 1 can be used to limit to a single job."
https://cmake.org/cmake/help/latest/man … ke-build-j
So what's the solution here? I want to build those packages as fast as possible with my system staying responsive while it's building, which means I want to limit the max number of parallel build jobs to the number of my CPU's max thread count, which works fine when building make-based projects and other non-cmake-based stuff.
I'm aware that cmake is a wrapper/generator for make (and other build tools), but why does it ignore the `MAKEFLAGS=-j32` then? Is it because it invokes multiple make commands under the hood in parallel, causing this wild "process expansion" without each individual make process being aware of this? Or is this dependent on the projects' CMakeLists.txt files themselves? As said in the beginning, I don't have much experience here.
Can the number of spawned build processes be limited in a different way without causing any failures? I've tried `systemd-run --user --pty --same-dir --wait -p TasksAccounting=1 -p TasksMax=32 makepkg ...`, but this results in "sh: fork: retry: Resource temporarily unavailable".
Thanks!
Offline
So what's the solution here? I want to build those packages as fast as possible with my system staying responsive while it's building
nice/ionice (latter depends on disk scheduler support)
Just to be clear, you're suggesting MAKEFLAGS, CMAKE_BUILD_PARALLEL_LEVEL and -j1 are *all* ignored?
Or does
Setting `-j1` on `cmake --build` does not do that, but that's obviously not the solution.
mean that constraining the build to a single job (*only*) prevents the overload? Did you try -j8 *there* as well?
Does that only affect
The problem is that when building kwin or plasma-workspace, my system always becomes super unresponsive and basically unusable. Since I don't have any problems when for example building the Linux kernel (also using my own PKGBUILD), I guess that the problem comes from cmake-based projects, like kwin and plasma-workspace.
?Did you try any other cmake based AUR build?
Process scheduling works perfectly fine and the system stays usable despite 100% CPU utilization.
…
It always results in 100% CPU utilization, so it looks like (way) more than 32 jobs are run in parallel, choking the CPU
Sanity check
It always results in 100% CPU utilization
for one core or all combined?
Offline
Does the cmake invocation include -G Ninja ?
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
Just to be clear, you're suggesting MAKEFLAGS, CMAKE_BUILD_PARALLEL_LEVEL and -j1 are *all* ignored?
I'm saying that despite `MAKEFLAGS=-j8` and `-j8` being set on the `cmake --build` command, more than 8 build jobs (way more than 32 build jobs actually) are run in parallel, taking 100% on all of my CPU cores and choking the system.
If `-j1` is set on the `cmake --build` command, only one build job is run. Hence why I think that cmake is running multiple make commands in parallel, which themselves are running multiple build jobs.
Did you try any other cmake based AUR build?
When building `strawberry` (or `clementine-git`) for example, the problem does not occur. Full utilization of all CPU cores, but the system stays responsive. Smaller projects also don't cause any issues.
So this looks like it's related to `kwin` and `plasma-workspace`. Btw, same problem when using the PKGBUILDs from the extra repo, so this is not caused by my modified PKGBUILDs.
Does the cmake invocation include -G Ninja ?
No, it does not on the `kwin` and `plasma-workspace` PKGBUILDs. The ninja build tool does not cause these issues. I've built `qt6-base` in order to check, which uses Ninja via cmake. Full CPU usage on all cores, no system responsiveness issues.
Offline
Full utilization of all CPU cores, but the system stays responsive. Smaller projects also don't cause any issues.
I've been maintaining my own kwin and plasma-workspace PKGBUILDs for a long time now
1. you should probably share those PKGBUILDs
2. have your tried to run this w/ ionice
3. Are you maybe running out of RAM and swapping a lot when building those projects (and need to drastically cut back on jobs to avoid that) since apparently the CPU charge (you're looking at all cores combined, not the utilization of one core only, are you?) isn't the limiting factor here?
Offline
I have already said that the same issue occurs with the PKGBUILDs from the extra repo. Here are the current PKGBUILDs of both packages.
https://gitlab.archlinux.org/archlinux/ … _type=tags
https://gitlab.archlinux.org/archlinux/ … _type=tags
This is not an I/O issue. I'm building on tmpfs (`BUILDDIR=/tmp/makepkg`). My system has 64 GiB of memory with >55 GiB available.
If these things were an issue, I'd surely have the same issues while building other/similar packages, which as said, is not the case.
Offline