Arch Linux

Skunky

Member

Registered: 2018-01-25

Posts: 231

[Solved] Can't get makepkg use all my CPU cores

Hello everyone i have an i73630QM, for some reason i can't get makepkg to use more than one core making build time pretty long, i'm following This guide to improve build times , i tried changing

MAKEFLAGS

option in /etc/makepkg.conf with

MAKEFLAGS="-j$(nproc)"

MAKEFLAGS="-j8" 

MAKEFLAGS="-j4"

with no success

Here is my current /etc/makepkg.conf

#!/hint/bash
#
# /etc/makepkg.conf
#

#########################################################################
# SOURCE ACQUISITION
#########################################################################
#
#-- The download utilities that makepkg should use to acquire sources
#  Format: 'protocol::agent'
DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u'
          'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u'
          'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u'
          'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u'
          'rsync::/usr/bin/rsync --no-motd -z %u %o'
          'scp::/usr/bin/scp -C %u %o')

# Other common tools:
# /usr/bin/snarf
# /usr/bin/lftpget -c
# /usr/bin/wget

#-- The package required by makepkg to download VCS sources
#  Format: 'protocol::package'
VCSCLIENTS=('bzr::bzr'
            'git::git'
            'hg::mercurial'
            'svn::subversion')

#########################################################################
# ARCHITECTURE, COMPILE FLAGS
#########################################################################
#
CARCH="x86_64"
CHOST="x86_64-pc-linux-gnu"

#-- Compiler and Linker Flags
CPPFLAGS="-D_FORTIFY_SOURCE=2"
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt"
CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"
#-- Make Flags: change this for DistCC/SMP systems
MAKEFLAGS="-j8"
#-- Debugging flags
DEBUG_CFLAGS="-g -fvar-tracking-assignments"
DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"

#########################################################################
# BUILD ENVIRONMENT
#########################################################################
#
# Defaults: BUILDENV=(!distcc !color !ccache check !sign)
#  A negated environment option will do the opposite of the comments below.
#
#-- distcc:   Use the Distributed C/C++/ObjC compiler
#-- color:    Colorize output messages
#-- ccache:   Use ccache to cache compilation
#-- check:    Run the check() function if present in the PKGBUILD
#-- sign:     Generate PGP signature file
#
BUILDENV=(!distcc color !ccache check !sign)
#
#-- If using DistCC, your MAKEFLAGS will also need modification. In addition,
#-- specify a space-delimited list of hosts running in the DistCC cluster.
#DISTCC_HOSTS=""
#
#-- Specify a directory for package building.
BUILDDIR=/tmp/makepkg

#########################################################################
# GLOBAL PACKAGE OPTIONS
#   These are default values for the options=() settings
#########################################################################
#
# Default: OPTIONS=(!strip docs libtool staticlibs emptydirs !zipman !purge !debug)
#  A negated option will do the opposite of the comments below.
#
#-- strip:      Strip symbols from binaries/libraries
#-- docs:       Save doc directories specified by DOC_DIRS
#-- libtool:    Leave libtool (.la) files in packages
#-- staticlibs: Leave static library (.a) files in packages
#-- emptydirs:  Leave empty directories in packages
#-- zipman:     Compress manual (man and info) pages in MAN_DIRS with gzip
#-- purge:      Remove files specified by PURGE_TARGETS
#-- debug:      Add debugging flags as specified in DEBUG_* variables
#
OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !debug)

#-- File integrity checks to use. Valid: md5, sha1, sha256, sha384, sha512
INTEGRITY_CHECK=(md5)
#-- Options to be used when stripping binaries. See `man strip' for details.
STRIP_BINARIES="--strip-all"
#-- Options to be used when stripping shared libraries. See `man strip' for details.
STRIP_SHARED="--strip-unneeded"
#-- Options to be used when stripping static libraries. See `man strip' for details.
STRIP_STATIC="--strip-debug"
#-- Manual (man and info) directories to compress (if zipman is specified)
MAN_DIRS=({usr{,/local}{,/share},opt/*}/{man,info})
#-- Doc directories to remove (if !docs is specified)
DOC_DIRS=(usr/{,local/}{,share/}{doc,gtk-doc} opt/*/{doc,gtk-doc})
#-- Files to be removed from all packages (if purge is specified)
PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod)
#-- Directory to store source code in for debug packages
DBGSRCDIR="/usr/src/debug"

#########################################################################
# PACKAGE OUTPUT
#########################################################################
#
# Default: put built package and cached source in build directory
#
#-- Destination: specify a fixed directory where all packages will be placed
#PKGDEST=/home/packages
#-- Source cache: specify a fixed directory where source files will be cached
#SRCDEST=/home/sources
#-- Source packages: specify a fixed directory where all src packages will be placed
#SRCPKGDEST=/home/srcpackages
#-- Log files: specify a fixed directory where all log files will be placed
#LOGDEST=/home/makepkglogs
#-- Packager: name/email of the person or organization building packages
#PACKAGER="John Doe <john@doe.com>"
#-- Specify a key to use for package signing
#GPGKEY=""

#########################################################################
# COMPRESSION DEFAULTS
#########################################################################
#
COMPRESSGZ=(gzip -c -f -n)
COMPRESSBZ2=(bzip2 -c -f)
COMPRESSXZ=(xz -c -z -)
COMPRESSLRZ=(lrzip -q)
COMPRESSLZO=(lzop -q)
COMPRESSZ=(compress -c -f)

#########################################################################
# EXTENSION DEFAULTS
#########################################################################
#
# WARNING: Do NOT modify these variables unless you know what you are
#          doing.
#
PKGEXT='.pkg.tar.xz'
SRCEXT='.src.tar.gz'

Any kind of help appreciated

Last edited by Skunky (2019-02-07 06:41:25)

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,531

Website

Re: [Solved] Can't get makepkg use all my CPU cores

What packages have you tried building with those settings?

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

Thanks for your reply,
I tried Vivaldi,android-studio,Waterfox and more, each of these didn't use more than one core, i tried to check PKGBUILD and seems like these packages don't use -j1 option

Last edited by Skunky (2019-02-06 15:36:31)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: [Solved] Can't get makepkg use all my CPU cores

None of those packages actually build anything.

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

None of those packages actually build anything.

Fixed aur links sorry

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: [Solved] Can't get makepkg use all my CPU cores

My comment had nothing to do with AUR links. They're package that download upstream binaries and repack them, there's no building (compiling) involved.

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

Sorry my mistake im still a newbie, i meant compressing package is very slow

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: [Solved] Can't get makepkg use all my CPU cores

That's covered in the same tips and tricks session you linked to.

loqs

Member

Registered: 2014-03-06

Posts: 19,106

Re: [Solved] Can't get makepkg use all my CPU cores

For locally built packages you could change the compression or removed compression by changing PKGEXT in makepkg.conf or the PKGBUILD.

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

That's covered in the same tips and tricks session you linked to.

Yes i modified makepkg parameters but that made no difference

For locally built packages you could change the compression or removed compression by changing PKGEXT in makepkg.conf or the PKGBUILD.

I removed

 PKGEXT='.pkg.tar.xz'

line but made no difference thanks everyone for your help

Last edited by Skunky (2019-02-06 16:03:04)

loqs

Member

Registered: 2014-03-06

Posts: 19,106

Re: [Solved] Can't get makepkg use all my CPU cores

You need to set PKGEXT to something other than the default e.g. PKGEXT='.pkg.tar' or PKGEXT='.pkg.tar.gz'

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

You need to set PKGEXT to something other than the default e.g. PKGEXT='.pkg.tar' or PKGEXT='.pkg.tar.gz'

Tried this

PKGEXT='.pkg.tar'

Still nothing i have no idea what it could be, it works fine on my other Arch setup

loqs

Member

Registered: 2014-03-06

Posts: 19,106

Re: [Solved] Can't get makepkg use all my CPU cores

The time to create the package was the same as with using xz compression?

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

The time to create the package was the same as with using xz compression?

I just made some benchmarks

PKGEXT='.pkg.tar.xz'

took 2 minutes and 31 seconds while

PKGEXT='.pkg.tar'

took 2 minutes and 54 seconds

Last edited by Skunky (2019-02-06 16:49:56)

loqs

Member

Registered: 2014-03-06

Posts: 19,106

Re: [Solved] Can't get makepkg use all my CPU cores

You could experiment with other compression options or using multiple cores on compression to try and find the best balance for your system.
Looking at the timings it would seem writing to disk on that system is expensive compared to compression.

Last edited by loqs (2019-02-06 16:58:32)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: [Solved] Can't get makepkg use all my CPU cores

That's covered in the same tips and tricks session you linked to.

Yes i modified makepkg parameters but that made no difference

No, you didn't. You stopped reading the tips and tricks when you got to the building section, which has nothing to do with what you're wanting, and skipped the compression part.

The time to create the package was the same as with using xz compression?

I just made some benchmarks

PKGEXT='.pkg.tar.xz'

took 2 minutes and 31 seconds while

PKGEXT='.pkg.tar'

took 2 minutes and 54 seconds

And the final package created was .pkg.tar?

Last edited by Scimmia (2019-02-06 17:15:56)

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

No, you didn't. You stopped reading the tips and tricks when you got to the building section, which has nothing to do with what you're wanting, and skipped the compression part.

You were right, now i added

--thread=0

to COMPRESSXZ option

And the final package created was .pkg.tar?

for some reason it wasn't, even if i specified this

PKGEXT='.pkg.tar'

in makepkg.conf, however now it finish in 10 seconds if i launch makepkg with

PKGEXT='.pkg.tar' makepkg

Thank you all very much for helping me out i would never made it alone, anyway, now i don't understand why my PKGEXT doesn't work

This is the exact line i'm using

PKGEXT='.pkg.tar'

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: [Solved] Can't get makepkg use all my CPU cores

Do you have PKGEXT set in the environment? Do you have $XDG_CONFIG_HOME/pacman/makepkg.conf ($XDG_CONFIG_HOME defaults to ~/.config/ IIRC) or ~/.makepkg.conf

spsf64

Member

From: FL, US or SP, BR

Registered: 2012-05-18

Posts: 109

Website

Re: [Solved] Can't get makepkg use all my CPU cores

@Skunky, as Loqs posted above for faster compression read: https://wiki.archlinux.org/index.php?ti … ompression

I edited my makepkg.conf to look like this:

COMPRESSXZ=(xz -c -T $(nproc) -z -)

This seems to kick all cpus to 100% when compressing to xz

---

Sergio S.
Yes, I'm old and bald!

Skunky

Member

Registered: 2018-01-25

Posts: 231

Re: [Solved] Can't get makepkg use all my CPU cores

Do you have PKGEXT set in the environment? Do you have $XDG_CONFIG_HOME/pacman/makepkg.conf ($XDG_CONFIG_HOME defaults to ~/.config/ IIRC) or ~/.makepkg.conf

I tried creating one and it worked, marking as solved thank you all!!