[Solved] System not using multithread / all CPUs for several tasks

Utini · 2023-12-19 14:29:25

Hello everyone,
I noticed that for some tasks my system doesn't utilize all CPUs / multithread even though it is configured to do so.
E.g. when building packages with paru or extracting/archiving files in .zip with Ark or peazip.
It will always use only 1 CPU at 100% while the other 15 CPUs are on idle.
I got 32GB RAM and also a PCI4 NVME with ~7.000MB/S read/write.

I wonder if I forgot to configure something or miss a package?

e.g. my /etc/makepkg.conf:

CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -D_FORTIFY_SOURCE=2 -fno-plt"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"
LTOFLAGS="-flto=auto"
RUSTFLAGS="-C opt-level=2 -C target-cpu=native"
MAKEFLAGS="-j16"
DEBUG_CFLAGS="-g"
DEBUG_CXXFLAGS="$DEBUG_CFLAGS"

BUILDDIR=/tmp/makepkg

COMPRESSZST=(zstd -c -z -q --threads=0 -)
COMPRESSXZ=(xz -c -z --threads=0 -)
COMPRESSGZ=(pigz -c -f -n)
COMPRESSBZ2=(pbzip2 -c -f)
COMPRESSLRZ=(lrzip -q)
COMPRESSLZO=(lzop -q)
COMPRESSZ=(compress -c -f)
COMPRESSLZ4=(lz4 -q)
COMPRESSLZ=(lzip -c -f)

PKGEXT='.pkg.tar'
SRCEXT='.src.tar.gz'

Additionally I have ark & peazip installed (also 7-zip) but the compression/extracting of .zip is also always only on 1 CPU.

Last edited by Utini (2023-12-19 16:47:58)

Scimmia · 2023-12-19 14:41:14

You've configured it not to compress the package, so the COMPRESS options don't do anything. Extraction is done with bsdtar, having ark and peazip installed means nothing. MAKEFLAGS is only going to affect makefiles, and not in all situations. Give a specific example.

Utini · 2023-12-19 14:53:06

Scimmia wrote:

You've configured it not to compress the package, so the COMPRESS options don't do anything. Extraction is done with bsdtar, having ark and peazip installed means nothing. MAKEFLAGS is only going to affect makefiles, and not in all situations. Give a specific example.

Yes that is true in terms of makepkg. I will try to give some examples:

Installing paru with paru itself will utilizie all CPUs during the building process (except for the last part 287/288 paru(bin) - this will again only use one CPU at 100%)
Choosing some random files with peazip or Ark (e.g. 2-3 folder with pictures and 2-3 10GB video files) and compressing them to .zip will only use one CPU.
Same for extracting this freshly created .zip archive.

7z a testarchive.zip -mmt16 testfiles: Uses one CPU
7z a testarchive.7z -mmt16 testfiles: Uses all CPUs
Extracing with a .zip 7z uses only one CPU

Can I give some better example or provide some better tests / logs somehow?

Last edited by Utini (2023-12-19 15:01:23)

V1del · 2023-12-19 15:08:06

.zip doesn't lend itself to multi-threaded extraction afaik and is unfixable in the format.(Don't have time to formally verify, so I might talk out of my ass but I'm pretty sure that was one of the limitations)

Last edited by V1del (2023-12-19 15:08:44)

Utini · 2023-12-19 16:41:50

V1del wrote:

.zip doesn't lend itself to multi-threaded extraction afaik and is unfixable in the format.(Don't have time to formally verify, so I might talk out of my ass but I'm pretty sure that was one of the limitations)

Hm maybe that is my problem (zip simply not being multithread compatible) and I am not actually fighting anything that is wrong.

I just tried it on windows with 7-zip.
.zip multithread support (using bzip2) is available there, although slower than single-core deflate.
Maybe it yields a higher compression rate with bzip2.

Peazip author replied to me on github:
zip currently has no multithread compression / decompression (or atleast no easy out-of-the-box way).
I will close this since everything seems to work as intended then.

Thanks!

Last edited by Utini (2023-12-19 16:47:45)

Arch Linux

#1 2023-12-19 14:29:25

[Solved] System not using multithread / all CPUs for several tasks

#2 2023-12-19 14:41:14

Re: [Solved] System not using multithread / all CPUs for several tasks

#3 2023-12-19 14:53:06

Re: [Solved] System not using multithread / all CPUs for several tasks

#4 2023-12-19 15:08:06

Re: [Solved] System not using multithread / all CPUs for several tasks

#5 2023-12-19 16:41:50

Re: [Solved] System not using multithread / all CPUs for several tasks

Board footer