[SOLVED] Pacman -Q can take a long time to execute

fdservices · 2017-06-11 07:19:24

I have a little script which calls

First: pacman -Qm; pacman -Qs
Second: pacman -Ss

Normally "First" executes in approximately half a second and "Second" in 0.56 seconds.

However, just occasionally "First" can take 15 seconds to complete, which presents a problem. My thought is that perhaps pacman is synchronising the package databases.

If that is so, how does pacman know that the databases are sufficiently out of sync to need to be updated, and how can I predict that the re-sync will be carried out?

If that is not the reason, then does anyone have any other suggestions?

Thanks in advance

Andrew

Last edited by fdservices (2017-06-17 14:28:47)

Allan · 2017-06-11 07:53:41

It is to do with the file cache. The first time pacman reads the files, the files are put into the system cache. The second time, the files are available in the cache and stuff happens fast. It is due to the local "database" being a number of small files.

fdservices · 2017-06-11 18:27:50

So it should only happen once after a start up and never a second time?

I will have to investigate now a command to update the cache without re-syncing.

Thanks for the info

Andrew

Trilby · 2017-06-11 18:53:17

Allan, it makes sense that a first query will take longer, but to me anything near 15 seconds seems indicative of additional problems, unless this is a very slow drive to start with (like a ridiculously old hard drive connected over a network file share).

V1del · 2017-06-11 20:46:05

To maybe clear up something, there's no network syncing happening, searching through your local files is what's taking a while. So you could just run a -Q command somewhere at the start of your session

ayekat · 2017-06-11 21:27:17

Try running with strace to see where it blocks:

strace pacman -Qm

--edit: V1del, running pacman -Qm at a session start is a workaround, but it would only be fighting the symptoms.

Last edited by ayekat (2017-06-11 21:29:03)

fdservices · 2017-06-13 07:42:43

Thanks for the help.

I shutdown the PC and restarted, and the first run took 16.5 seconds. The second run took the expected 0.5 seconds. So it looks as if the analysis is correct.

The PC is not that old but I will run some test on it anyway.

I take it that there is no pacman --updatecache command?

Andrew

twelveeighty · 2017-06-13 13:49:33

fdservices wrote:

Thanks for the help.
I take it that there is no pacman --updatecache command?

I think that Allen said pacman "uses" the system cache, which would mean running that command the 2nd time is faster because the system is reading from disk cache (in RAM), rather than your actual drive. If you're looking to make this command go faster by "pre-caching" during off hours, I guess simply running that command in a Cron and hope the disk cache isn't flushed by some other disk activity, that could work..?

progandy · 2017-06-13 14:46:46

twelveeighty wrote:

If you're looking to make this command go faster by "pre-caching" during off hours, I guess simply running that command in a Cron and hope the disk cache isn't flushed by some other disk activity, that
could work..?

You could also try installing vmtouch and use it to permanently mmap/mlock all files of your pacman database. That may waste up to 100MB of your RAM, but you will have quicker pacman database access. You may have to restart the locking after a package update to replace the old locked files with the new.

Last edited by progandy (2017-06-13 14:50:03)

fdservices · 2017-06-13 18:14:28

I think that Allen said pacman "uses" the system cache, which would mean running that command the 2nd time is faster

I understand that, I was looking to possibly control when the cache was created the first time, so that my script was not affected by this extra load when it runs. In any case, I don't think that the pacman command exists.

Can anyone time how long the caching process takes on their PC?

Thanks

Andrew

ayekat · 2017-06-13 18:17:42

fdservices wrote:

Can anyone time how long the caching process takes on their PC?

It does by far not take 15 seconds. That's why Trilby suggested that something might be odd, and I suggested that you could try to check with `strace` to see where exactly it blocks.

(unless you're running some very old hard drive)

Slithery · 2017-06-13 18:45:53

Only takes a couple of seconds for me, and this is on around 10 year old hardware with 5400RPM spinning rust in a slow array.

15 seconds sounds way off to me too.

apg · 2017-06-14 01:11:35

slithery wrote:

Only takes a couple of seconds for me, and this is on around 10 year old hardware with 5400RPM spinning rust in a slow array.
15 seconds sounds way off to me too.

How long it takes to load the database depends in significant part on how spread out the files are. I've had load times in the 15 second range before with nothing amiss. If you group the files together on the disk and it still takes a long time, then I would worry that something is wrong.

fdservices wrote:

I think that Allen said pacman "uses" the system cache, which would mean running that command the 2nd time is faster
I understand that, I was looking to possibly control when the cache was created the first time, so that my script was not affected by this extra load when it runs. In any case, I don't think that the pacman command exists.

pacman doesn't need a command to update the cache, because pacman has nothing to do with the cache. Just run any pacman command that reads the local database somewhere it will get executed before your script.

Xyne · 2017-06-14 01:35:51

The 15 seconds is intentional. It's Allan stealing your life away.

Have you tried running pacman-optimize?

pacman-optimize--help wrote:

pacman-optimize (pacman) 5.0.1
Usage: /usr/bin/pacman-optimize [--nocolor] [pacman_db_root]
pacman-optimize is a little hack that should improve the performance
of pacman when reading/writing to its filesystem-based database.
Because pacman uses many small files to keep track of packages,
there is a tendency for these files to become fragmented over time.
This script attempts to relocate these small files into one
continuous location on your hard drive. The result is that the hard
drive should be able to read them faster, since the hard drive head
does not have to move around the disk as much.

Allan · 2017-06-14 03:26:54

pacman-optimize is placebo. All files end up in your cache afterwards, so it looks fast...

Xyne · 2017-06-14 04:41:08

Allan wrote:

pacman-optimize is placebo. All files end up in your cache afterwards, so it looks fast...

I hesitated to mention it. I understand the reasoning behind the poor man's defrag but I don't know enough about disk performance or the effectiveness of the tar+untar approach to evaluate it. I figured that it can't hurt to try and it would be interesting if it had an impact on loading time the next time the files are uncached (obviously not right after running the command).

fdservices · 2017-06-14 10:24:25

Thanks for that, pacman-optimize seems to be exacty what I need. Unfortunately after runnng this the script took 25 seconds to run. :-(

So I checked further and -Qm seems to run quite quickly but -Qs takes a long time, which does not mesh with the above:

Qm 415281 microseconds per iteration
Qs 22871429 microseconds per iteration

These times are skewed because there are timers running on them and a strace on -Qs

This is the strace output

I am now throroughly confused.

Andrew

Trilby · 2017-06-14 10:35:18

While I still think that length of time is indicative of another problem, your most recent test doesn't actually say much. Pacman -Qs accesses a completely different set of files than pacman -Qm. Pacman -Qs needs to read every package desc file while -Qm does not. In fact your -Qs needs to read 1417 different desc files. So if each one takes even 0.01 seconds, that would account for your 15 seconds.

Your strace seems fine. It's doing all the right steps, and only the right steps, so it does seem that it is just slow file access. Have you run any SMART or similar disk diagnostics?

fdservices · 2017-06-14 13:42:04

That is all correct, except that the second run is as follows:

Qm 414604 microseconds per iteration
Qs 63817 microseconds per iteration

which means that it is the first run of -Qs which takes the time, and after that it is very quick, whereas -Qm is broadly the same on both runs. So can I assume that -Qs caches the database and -Qm does not? Is there a better -Q command to cache the database?

I will run some smart tests this afternoon

Andrew

Trilby · 2017-06-14 13:58:27

Yes, the first -Qs is the first time all the desc files are accessed (regardless of how many other pacman commands you have run that don't access the desc files). The second run of -Qs is faster for precisely the reasons given at the start of this thread: the desc files are cached.

As far as assuming that -Qs caches the database and -Qm does not, that is a faulty assumption at two levels. First, pacman doesn't cache in this sense, the kernel/filesystem does: any file access (pacman or not) will be faster the second time than the first. The second error is that there is not one singular "pacman database", there are just a bunch of files; different pacman commands access different subsets of these files.

The fact that -Qm is not faster the second time could be due to many factors. Most notable, though, is that it would not be expected to gain much: it only accessess a small number of files. Hypothetically, if system file caching provided a benefit of around 8ms then the change between the first and second -Qm command due to this caching might still only be a few dozen milliseconds while the change for -Qs could be nearly 12 seconds.

fdservices · 2017-06-15 07:22:30

SMart has not shown up any errors in the short test.

Just one further question of I may.

If the cache is created by the "system", then, presumably, it could be cleared by the system if it is calculated that the memory is needed by another process? While this might be unusual, if it is a pssibility, then it affects how I proceed.

May I thank you all, once again, for all your help.

Andrew

V1del · 2017-06-15 08:13:38

Correct. A quick googler has lead me to https://hoytech.com/vmtouch/ which is something you might be interested in, there's also an AUR package. Haven't tried it, but you can use it to lock files into the cache.

fdservices · 2017-06-15 08:21:50

Thank you. I am working on a solution to my problem

Andrew

Xyne · 2017-06-16 01:00:02

fdservices wrote:

Thanks for that, pacman-optimize seems to be exacty what I need. Unfortunately after runnng this the script took 25 seconds to run. :-(

I'm dying here. A tool named "pacman-optimize" that de-optimizes pacman. F@$%in' Allan strikes again!

Allan · 2017-06-16 04:01:15

I'm the one who deleted it from the repo!

Arch Linux

#1 2017-06-11 07:19:24

[SOLVED] Pacman -Q can take a long time to execute

#2 2017-06-11 07:53:41

Re: [SOLVED] Pacman -Q can take a long time to execute

#3 2017-06-11 18:27:50

Re: [SOLVED] Pacman -Q can take a long time to execute

#4 2017-06-11 18:53:17

Re: [SOLVED] Pacman -Q can take a long time to execute

#5 2017-06-11 20:46:05

Re: [SOLVED] Pacman -Q can take a long time to execute

#6 2017-06-11 21:27:17

Re: [SOLVED] Pacman -Q can take a long time to execute

#7 2017-06-13 07:42:43

Re: [SOLVED] Pacman -Q can take a long time to execute

#8 2017-06-13 13:49:33

Re: [SOLVED] Pacman -Q can take a long time to execute

#9 2017-06-13 14:46:46

Re: [SOLVED] Pacman -Q can take a long time to execute

#10 2017-06-13 18:14:28

Re: [SOLVED] Pacman -Q can take a long time to execute

#11 2017-06-13 18:17:42

Re: [SOLVED] Pacman -Q can take a long time to execute

#12 2017-06-13 18:45:53

Re: [SOLVED] Pacman -Q can take a long time to execute

#13 2017-06-14 01:11:35

Re: [SOLVED] Pacman -Q can take a long time to execute

#14 2017-06-14 01:35:51

Re: [SOLVED] Pacman -Q can take a long time to execute

#15 2017-06-14 03:26:54

Re: [SOLVED] Pacman -Q can take a long time to execute

#16 2017-06-14 04:41:08

Re: [SOLVED] Pacman -Q can take a long time to execute

#17 2017-06-14 10:24:25

Re: [SOLVED] Pacman -Q can take a long time to execute

#18 2017-06-14 10:35:18

Re: [SOLVED] Pacman -Q can take a long time to execute

#19 2017-06-14 13:42:04

Re: [SOLVED] Pacman -Q can take a long time to execute

#20 2017-06-14 13:58:27

Re: [SOLVED] Pacman -Q can take a long time to execute

#21 2017-06-15 07:22:30

Re: [SOLVED] Pacman -Q can take a long time to execute

#22 2017-06-15 08:13:38

Re: [SOLVED] Pacman -Q can take a long time to execute

#23 2017-06-15 08:21:50

Re: [SOLVED] Pacman -Q can take a long time to execute

#24 2017-06-16 01:00:02

Re: [SOLVED] Pacman -Q can take a long time to execute

#25 2017-06-16 04:01:15

Re: [SOLVED] Pacman -Q can take a long time to execute

Board footer