[Solved] omitting !debug increases memory & space used drastically

Lone_Wolf · 2024-02-26 13:26:57

I like to build my aur packages semi-regularly in a clean chroot to verify dependencies , build issues and such .

Lately this gives me problems due to lack of space . My / partition uses ext4, has 30+ Gib space free before a chroot is created, but that amount decreases fast while building with pkgctl / extra-x86_64-build / multilib-build & friends.

atm I'm trying to build my llvm-minimal-git and free space on root is now less than 1 Gib according to df -h .

My ~/.bashrc has export TMPDIR=/data/panoramix/tmp in it, but the drive that is on sees no change in free space during build .

Ah, the build just failed with 'no space left on device' .
df -h (as user) indicates there is 2.7M available.

removing /var/lib/archbuild/extra-x86_64 shows 34 Gib available .

Have others encountered such issues and could this be a bug in devtools ?

how can I redirect /var/lib/archbuild to another location ?

Last edited by Lone_Wolf (2024-03-20 14:12:28)

swsnr · 2024-02-26 16:40:13

Try a bind-moun onto an existing filesystem, or just a direct mount of another filesystem?

Preferably a btrfs filesystem, which allows devtools to use copy-on-write snapshots which consums a lot less disk space.

Last edited by swsnr (2024-02-26 16:41:59)

Lone_Wolf · 2024-02-27 15:33:51

I didn't need a bind mount, the -c <dir> option of extra-x86_64-build (& makechrootpkg) allowed to put the chroot on a different place with more free space .

Unfortunately this didn't solve the problem as the oom killer aborted the build.

Meanwhile running makepkg on the live system with the same PKGBUILD doesn't show this huge decrease in free space or the abortion .

Since my /etc/makepkg.conf limits the number of jobs to 75% of available cores (MAKEFLAGS="-j 18") I'm now testing extra-x86_64-build with lower job numbers hardcoded in the PKGBUILD. (currently testing with MAKEFLAGS="-j 12" )

the free space decrease is still present, and the build aborts around the same point where it did with no limit to jobs.

I'm certain I didn't have this issue early in january, so I'm going to downgrade devtools to the previous version , 1.1.04-1 from sept 2023 to test.

ADDED

oom killer interferes also with devtools 1.1.04-1 , so the cause must lie somewhere else.

Last edited by Lone_Wolf (2024-02-27 16:39:19)

Lone_Wolf · 2024-03-09 14:51:59

I have made progress.

Using my own (customized) /etc/pacman.conf & /etc/makepkg.conf with a classic chroot everything works fine just like makepkg -Crs as user.

Combining the same files with aur clrean-chroot-manager from graysky also works fine .

pkgctl build aborted when /var/lib/archbuild needed diskspace increased to above 25 Gib and doesn't appear to have an option to build in a different location (with more space) .
Going to test extra-x86_64-build (which does allow that) next.

Last edited by Lone_Wolf (2024-03-09 14:55:06)

Lone_Wolf · 2024-03-09 16:51:54

extra-x86_64-build -c -r /data/panoramix/chroot/ just finished .
time needed was 110 minutes, typical time for this package (with ccache disabled) is around 60 minutes.

some numbers :

# du -s /data/panoramix/chroot/root
964548  /data/panoramix/chroot/root
# du -s /data/panoramix/chroot/panoramix
9659688 /data/panoramix/chroot/panoramix
# du -s /data/panoramix/chroot/extra-x86_64/
88931916        /data/panoramix/chroot/extra-x86_64/
#

The first 2 are the sizes of the folders used by my classic chroot, the almost 10 times as big one is for the extra-x86_64 build !

During investigating I increased my swapspace from 8 to 24 GiB .
Some fun results of free -m during the extra-x86_64-build :

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845       13832        1108          12        1339        2013
Swap:          24575         917       23658
$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845       11551        3266          12        1461        4293
Swap:          24575         917       23658
$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845       15595         209           5         423         250
Swap:          24575       12375       12200
$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845       15389         409          10         439         455
Swap:          24575       24266         309
$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845        3321        8211          16        4662       12523
Swap:          24575         981       23594
$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845       15570         173           4         544         275
Swap:          24575        6021       18554
$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15845        2020        5641          16        8533       13824
Swap:          24575         527       24048
$

Time to do other things, will try using my custom /etc/makepkg.conf with extra-x86_64-build in the coming days.

Lone_Wolf · 2024-03-19 15:02:28

I found the cause of the memory & space issues during build and changed the title of the thread.

The custom makepg.conf-file I use has !lto in OPTIONS.
The PKGBUILD has options=('!lto!) which according to man PKGBUILD is correct to disable lto for this package.

Unfortunately this doesn't work and the pacakge is build with lto which drastically increases free space & memory used .

Using OPTIONS=('!lto') in the PKGBUILD does prevent lto building as desired.

Man makepkg.conf uses all caps for env vars and man PKGBUILD lists lowercase env vars only, this causes confusion about the correct usage.

Should I file a bug report ?
And if so, should it be created against the archlinux pacman package or pacman upstream ?

Please advise.

Last edited by Lone_Wolf (2024-03-19 15:03:04)

WorMzy · 2024-03-19 15:17:04

Mod note: moving to Pacman issues.

Allan · 2024-03-19 22:03:12

Lone_Wolf wrote:

I found the cause of the memory & space issues during build and changed the title of the thread.
The custom makepg.conf-file I use has !lto in OPTIONS.
The PKGBUILD has options=('!lto!) which according to man PKGBUILD is correct to disable lto for this package.
Unfortunately this doesn't work and the pacakge is build with lto which drastically increases free space & memory used .
Using OPTIONS=('!lto') in the PKGBUILD does prevent lto building as desired.
Man makepkg.conf uses all caps for env vars and man PKGBUILD lists lowercase env vars only, this causes confusion about the correct usage.
Should I file a bug report ?
And if so, should it be created against the archlinux pacman package or pacman upstream ?
Please advise.

This is the wrong assessment. There are many packages using options=('!lto') in the repos, and that disables LTO.

What you are doing with OPTIONS=('!lto') in the PKGBUILD is overriding the OPTIONS= line in makepkg.conf. My guess is that stops a debug package being built - for some reason Arch enabled that by default. Try adding "!debug" to the options in the PKGBUILD.

Lone_Wolf · 2024-03-20 14:10:23

Further testing confirmed it was not lto but debug that caused the issue.
Marking as Solved.

My custom makepkg.conf uses

BUILDENV=(!distcc color ccache check !sign)
OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !debug !lto)

I'm going to check all my aur packages to ensure they explicitly enable/disable lto & debug to avoid this issue in the future.

Thanks for the input, Allan .

Arch Linux

#1 2024-02-26 13:26:57

[Solved] omitting !debug increases memory & space used drastically

#2 2024-02-26 16:40:13

Re: [Solved] omitting !debug increases memory & space used drastically

#3 2024-02-27 15:33:51

Re: [Solved] omitting !debug increases memory & space used drastically

#4 2024-03-09 14:51:59

Re: [Solved] omitting !debug increases memory & space used drastically

#5 2024-03-09 16:51:54

Re: [Solved] omitting !debug increases memory & space used drastically

#6 2024-03-19 15:02:28

Re: [Solved] omitting !debug increases memory & space used drastically

#7 2024-03-19 15:17:04

Re: [Solved] omitting !debug increases memory & space used drastically

#8 2024-03-19 22:03:12

Re: [Solved] omitting !debug increases memory & space used drastically

#9 2024-03-20 14:10:23

Re: [Solved] omitting !debug increases memory & space used drastically

Board footer