[Solved] pacman -F /-Fx memory consumption - alternative ways?

GerBra · 2025-12-31 00:10:59

Hello,

i have to query a file or part of a file to which package it belongs. The "search" must also be done for packages not already installed on the system.

Normally a wonderful job for pacman -F or -Fx. The results are always correctly and well done.
So what's my problem?

One process consumes a lot of memory doing this. On a half-way modern system there is no problem.
But currently i try such a search on an older Raspi Pi, and it takes the system to "no responding" anymore. I work via ssh on the device, a scheduled reboot shows since around 20min that it reboots NOW <g>

- First i thought it may be a IO Problem (only a sd card)
So i moved the sync/<repo>.files dir to /dev/shm and use this dir as --dbpath. No solution.

- nice'ing also no solution

- stracing/--debug options on pacman shows: the problem starts when it comes to our BIG extra.files. Around 50MB size and pacman loads information for around 14500 packages from. extra.files get opened, parsed for a while and then stucks reading therein.
So i assume it's a memory resource problem on my device. Last what i could see via htop/top is that the pacman process consumes nearly 500MB RES.
My raspi (take in any other low memory hardware, must not be ARM) has only 1G RAM, and there are other processes running.

- i tried also a approach with pacsift, but it's the same (guess it uses the same lib-routines when working wit <repo>.files DB)

- if i remove the extra.files from the --dbpath, than both (pacman /pacsift) finished flawless, cause only working with core.files, multilib.files and my local repo.files (not as big as extra.files)

- I will try another run with swap(file) enabled, currently no swap is active.

My question: <g>
Do you know any other way/idea to query a package belonging to a <search term> without using pacman/pacsift? Something "remote"? A website-API? A other tool? grep/awk directly in <repo>.files ?

//Edit: Read wiki<g> -> found that maybe pkgfile is a alternative, will try it...

Thank's for reading, and a nice new year...

Solution: pkgfile does the job

Wiki wrote:

For advanced functionality, install pkgfile, which uses a separate database with all files and their associated packages.

pkgfile -s -r brcmfmac43012-sdio.clm_blob
core/linux-firmware-broadcom

Tip: Still when you mean that you know "everything"<g>, it's a good idea to look at the wiki before bother others...

Last edited by GerBra (2025-12-31 00:44:50)

system72 · 2025-12-31 00:59:38

i can also see about 800mb of ram being consumed when executing this:

pacman -Fx 'pacman|hyprland|ls|cp|mv|rename|lsd|bat|cat|nvim|shellcheck'

maybe its a memory problem with pacman itself

GerBra · 2025-12-31 01:09:52

system72 wrote:

maybe its a memory problem with pacman

Maybe, but since community+extra were merged searching this massive files may also show a not so good search algorithm.

pkgfile - AFAIS - uses more splitted <repo>."files", maybe other DB format and runs multi-threaded.

I currently work on a little script for detecting needed firmware packages based on loaded modules:
https://codeberg.org/GerBra/detect_firmware.git
Diskussion(only german): https://forum.archlinux.de/d/35687-dete … t-probleme
where i use pacman -Fx very intensive.
Maybe i switch to pkgfile completely.

Last edited by GerBra (2025-12-31 01:13:36)

Allan · 2025-12-31 10:58:25

pacman is designed to read the entire sync database at once as the full information is needed for the update. This was carried over into the reading of the files database (which are an extension of the sync dbs). We should stream it like pkgfile does, but I never got around to dealing with that. I'm fairly sure there is a patch somewhere.

pkgfile also does some decompression/recompression of the databases on update, changing to a format that is faster to read.

GerBra · 2025-12-31 10:59:56

I think in this special case(low memory) the reason that pkgfile runs and exit properly, and pacman -F/-Fx not is:
pacman

$ file /var/lib/pacman/sync/extra.files
/var/lib/pacman/sync/extra.files: gzip compressed data, from Unix, original size modulo 2^32 554342400

pkgfile

[gerhard@ws01 ~]$ file /var/cache/pkgfile/extra.files.*
/var/cache/pkgfile/extra.files.000: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.001: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.002: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.003: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.004: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.005: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.006: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.007: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.008: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.009: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.010: ASCII cpio archive (SVR4 with no CRC)
/var/cache/pkgfile/extra.files.011: ASCII cpio archive (SVR4 with no CRC)

pacman's extra.files is compressed, uncompressed it's size is 554.342.400 Bytes (~555 MB)
So a pacman -Fx process - when it comes to extra - it must in memory decompress the file and allocate memory for all information's in this data size. Then performing the <search>

pkgfile splits extra into several "pieces" (uncompressed), and therein are only data information "pkgname/package_files". So pkgfile could do only one job - query a package_file and give the pkgname.

So pkgfile needs to allocate lot lesser memory for its process, cause: smaller files, no decompress, lesser data to parse.
And it seems that part of this process are separated into several threads.

That's IMHO why pkgfile is the "better"(?) tool for: find a package to which a <search> match.

//Edit: Beaten by Mr. pacman ;-)

Last edited by GerBra (2025-12-31 11:13:56)

lilydjwg · 2025-12-31 14:20:03

I have written a plocate wrapper program pacfiles that builds plocate databases and searches those instead. It's both faster (than pkgfile!) and resource efficient, plus retaining the same arguments and output format from pacman -F.

Last edited by lilydjwg (2025-12-31 14:20:33)

Arch Linux

#1 2025-12-31 00:10:59

[Solved] pacman -F /-Fx memory consumption - alternative ways?

#2 2025-12-31 00:59:38

Re: [Solved] pacman -F /-Fx memory consumption - alternative ways?

#3 2025-12-31 01:09:52

Re: [Solved] pacman -F /-Fx memory consumption - alternative ways?

#4 2025-12-31 10:58:25

Re: [Solved] pacman -F /-Fx memory consumption - alternative ways?

#5 2025-12-31 10:59:56

Re: [Solved] pacman -F /-Fx memory consumption - alternative ways?

#6 2025-12-31 14:20:03

Re: [Solved] pacman -F /-Fx memory consumption - alternative ways?

Board footer