You are not logged in.

#26 2017-06-16 04:49:36

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

Allan wrote:

I'm the one who deleted it from the repo!

Yeah, after someone benchmarked it. Brilliant! big_smile
I'll be waiting for the "fixmysystem" tool next tongue


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#27 2017-06-17 12:19:17

twelveeighty
Member
From: Alberta, Canada
Registered: 2011-09-04
Posts: 1,096

Re: [SOLVED] Pacman -Q can take a long time to execute

V1del wrote:

Correct. A quick googler has lead me to https://hoytech.com/vmtouch/ which is something you might be interested in, there's also an AUR package. Haven't tried it, but you can use it to lock files into the cache.

There's also a companion package called vmtouch-systemd (https://aur.archlinux.org/packages/vmtouch-systemd-git/) that launches it as a daemon. Its default configuration file locks /var/lib/pacman in cache. Haven't tried it myself, but in theory this does exactly what OP wants.

Edit: the other deamon.

Last edited by twelveeighty (2017-06-17 12:20:50)

Offline

#28 2017-06-17 14:27:17

fdservices
Member
From: France
Registered: 2012-02-06
Posts: 413

Re: [SOLVED] Pacman -Q can take a long time to execute

I am going to mark this as solved, not because it is, but because it seems that there is no real solution.

My understanding now is that the local database [sic] (taken from the pacman man pages) is not particularly managable. It is a series of folders and files, which have to be opened, read and then closed to establish the content. Even then they are all labelled as "local" packages because the source repository is not a part of the data saved in the files. Assuming that local packages are those installed locally, i.e. from AUR or from another PKGFILE, then that means that it becomes quite difficult to establish which packages have been installed locally (pacman -Qm takes 0.5 seconds on my PC to find less than 50 packages) . Equally to simply list all the details of the packages which have been installed on the PC using pacman will normally take some time (pacman -Qs takes 15+ seconds on my PC to list around 1,400 packages).

Now the results of pacman -Qs will be held in memory (cached) for future use, meaning that it will normally appear to execute very quickly after the first time that it is run, assuming that one needs to run it more than once.

My solution has been not to use pacman -Qs at all, and to use pacman -Qm alone to establish the locally installed packages. This has actually saved .02 seconds in the original execution time of my script even when the -Qs command had already been cached.

I consider this an excellent outcome, made possible only by all the help and advice in the above replies, even though the "solution" is not to go by that route at all because it is not a very well constructed road :-)

Thanks again to all

Andrew

edit 18/07/2017 for factually inaccuracy and clarification

Last edited by fdservices (2017-06-18 07:49:53)

Offline

#29 2017-06-17 14:33:12

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [SOLVED] Pacman -Q can take a long time to execute

All of that shows you have no understanding of anything at all. It is not .tar.xz files, you don't know what "installed locally" means, and the results are not stored in memory.

Offline

#30 2017-06-17 15:01:25

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

Scimmia wrote:

All of that shows you have no understanding of anything at all.

I was about to start with a very similar comment.  fdservices, you seem to repeatedly be very quick to jump to faulty conclusions.  This will hinder any potential for learning.  It is ok to not know how things work, but your ridiculous assumptions only serve to maintain that ignorance - this is less ok (really, it's your business, but if you seek help here you should try to be more open to learning).

I'd like to try to offer some corrections to your last summary, but it is just so completely wrong I wouldn't know where to start.  Suffice it to say that if you want to understand more about pacman you are headed in the wrong direction.  Stop and start again.

Also, it seems you are using pacman quite wrong.  You've not yet described what the purpose is of these commands you are running, but you've now just revealed that one of them is completely the wrong command.  If you want to list all the installed packages, use `pacman -Q` not `pacman -Qs`.  The 's' is for searching for a match and really only makes sense when provided with a search string.  Searching for a match requires reading every single .desc file, simply listing installed packages does not.  `pacman -Q` will be much faster than `pacman -Qs` - the difference is small (but measurable) on my SSD system, but on your apparently slow hard drive, you should see a substantial difference.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#31 2017-06-17 18:02:00

fdservices
Member
From: France
Registered: 2012-02-06
Posts: 413

Re: [SOLVED] Pacman -Q can take a long time to execute

i am not sure where the misunderstandings have arisen. /var/cache/pacman/pkg seems to be full of .tar.xv files may be this is not the data that Allan was referring to? Also, he, and others, stated that the results were cached (held in RAM) after the first read. This seemed to fit the observations. p.s I am not sure why you use the term "stored".

@Trilby Feel free to criticize, Where have I gone wrong?

If you want to list all the installed packages, use `pacman -Q` not `pacman -Qs`

quite - I did not want a simple list of installed packages, I wanted the descriptions and other data as well, for all the installed packages. It seems you may have jumped to a conclusion?

In any case, with your help, I found an alternative solution which works.

Thank you

Andrew

Offline

#32 2017-06-17 18:32:18

adesh
Member
Registered: 2016-10-05
Posts: 167

Re: [SOLVED] Pacman -Q can take a long time to execute

@fdservices
Files stored in /var/cache/pacman/pkg are the files that pacman downloads when installing package. This is just cache and you can delete (although not recommended) these files without any issues, you will not break you system in any way. These files do not represent the database that pacman deals with when using -Q (as in your case) and -S options. Pacman local database (for packages installed on you system) is stored in /var/lib/pacman/local, sync database (all packages available to install) in /var/lib/pacman/sync, and that is what Trilby was talking about.

A better way to get information about locally installed packages would be: pacman -Qi
Please read more about pacman on the wiki.

Offline

#33 2017-06-17 19:22:07

fdservices
Member
From: France
Registered: 2012-02-06
Posts: 413

Re: [SOLVED] Pacman -Q can take a long time to execute

Thank you, I am grateful. That clears up a couple of points. Unfortunately pacman -Qi is equally slow to complete on my PC.

Andrew

Offline

#34 2017-06-17 19:25:30

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [SOLVED] Pacman -Q can take a long time to execute

Nobody said the results were cached. Read up on what disk cache is.

Offline

#35 2017-06-17 19:36:06

fdservices
Member
From: France
Registered: 2012-02-06
Posts: 413

Re: [SOLVED] Pacman -Q can take a long time to execute

@Allan

The second time, the files are available in the cache

@Webopedia

Disk cache: A portion of RAM used to speed up access to data on a disk.

Andrew

Offline

#36 2017-06-17 19:43:50

ayekat
Member
Registered: 2011-01-17
Posts: 1,589

Re: [SOLVED] Pacman -Q can take a long time to execute

The cache in memory has nothing to do with cached files on the disk (although the basic idea is the same).

/var/cache is simply a place where applications may cache their data, persistently.
The memory cache is data that is held in memory to avoid having to read it from the disk.


pkgshackscfgblag

Offline

#37 2017-06-18 00:39:23

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

Ok, here we go. Gather around.

Files are stored as data on a hard drive. Two files in the same directory on your computer may be stored in completely different areas on the disk, the same way two people next to each other in a phone book may live miles apart. When you open a file, the computer tells the hard drive to retrieve it. If you request multiple files, each one must be retrieved. If those files are next to each other on the disk, they can be retrieved quickly in one read. If they are all over the disk, then the hard drive has to read one file, then find the next one, then read it, then find the next one, etc. Jumping around on the disk takes time. Btw, this is why defragging a disk can speed it up (fun fact: even single files can be broken up and stored in different areas of the disk).

The "cache" here is the page cache or "disk cache". When you request a file from the hard drive, the operating system may store it in memory temporarily. If the file is requested again without having changed, it can be read straight from memory, which is much faster. It's like walking into a library to ask for a book that you've previously checked out and the librarian has it waiting on the desk instead of on some obscure bookshelf.

/var/cache/pacman/pkg is a "cache" of downloaded packages. It has nothing to do with anything in this thread. "cache" is a general term. In computing, it usually means temporary storage of some resource that requires overhead to acquire, to avoid that overhead if it's needed again in the near future.

The local pacman database is in /var/lib/pacman/local. It's just a hierarchy of regular files which can easily number in the thousands. Pacman has to open them whenever it retrieves info from the local database. That takes time. You can actually blame Allan for that. Once they have been read, they are usually held in the page cache so subsequent reads are faster. They might not be cached though. The purpose of vmtouch is supposedly to keep those files in the cache. Basically, it's like a memo to the librarian to keep some books on the desk so you can walk in and read them right away whenever you feel like it. The cache has a limited size though, so that might slow down other operations (you can't keep all the books in the library on the librarian's desk).

The sync databases are single archives so they are much faster to read (you can thank Allan for that).

You're also wrong about how pacman -Q* works. pacman -Qm creates a list of all packages in the local database, then removes all packages that are found in the sync database. Everything left over is "foreign", i.e. pacman has no idea where it came from. The output of pacman -Qm, -Qi, or -Q anything is not stored in memory. It reads the files in the local database every time. It's just faster after the first time because the files have been cached in the page cache. Running pacman -Qi should force pacman to read all information from the local database and thus (page) cache the local database files, which should make everything faster on subsequent runs.

Is that clear(er)?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#38 2017-06-18 07:01:47

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

Xyne wrote:

You can actually blame Allan for that.

Decisions made long before my time...  I just have not done anything to improve it!


Xyne wrote:

The sync databases are single archives so they are much faster to read (you can thank Allan for that).

Tar based backends are awesome!

Offline

#39 2018-01-22 06:07:26

wsha
Member
Registered: 2015-07-22
Posts: 19

Re: [SOLVED] Pacman -Q can take a long time to execute

I have the same symptoms as fdservices. "pacman -Syu" takes about 20 seconds to run after "sync; echo 2 | sudo tee /proc/sys/vm/drop_caches" and about 1 second to run if I run it again immediately. I can't say for sure when it got slow, but I remember it being faster before. I am using a 5400 RPM HDD that is about 3 years old. I don't see anything wrong in SMART. Looking at strace, I don't see one call blocking for a long time. During the slow part ("resolving dependencies..." and "looking for conflicting packages...") I see lots of calls to open and read the desc files in /var/lib/pacman/local and don't see any single command blocking for significant time. "pacman -Q" is 1378 lines long, and "pacman -Qqet" is 149.

Does 20 seconds seem like an excessive amount of time for this set up? I feel like I have a pretty stripped down system (for a graphical desktop). I tried pacman-optimize but saw no improvement.

Offline

#40 2018-01-22 06:36:22

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

20sec sounds about right.  This is pure file access times.  None taking a long time, but there are ~1400 files being accessed on your system.

I recently upgraded my laptop to one with a SSD.  That wait has now completely gone.

Offline

#41 2018-01-22 14:07:00

apg
Developer
Registered: 2012-11-10
Posts: 211

Re: [SOLVED] Pacman -Q can take a long time to execute

You can try this slightly modified version of pacman-optimize.

Offline

#42 2018-01-26 12:30:27

wsha
Member
Registered: 2015-07-22
Posts: 19

Re: [SOLVED] Pacman -Q can take a long time to execute

Wow, apg, that modified version does seem to have helped. My access time has dropped from 20 seconds to 4-5 seconds. How does that modification make a difference? Diffing against /usr/bin/pacman-optimize, the only change I see is that

bsdtar -czf "$workdir/pacman-db.tar.gz" -C "$localdb" ./

becomes

(cd "$localdb" && bsdtar -czf "$workdir/pacman-db.tar.gz" ./*)

which looks like it should be exactly the same.

I had been thinking about why I perceived pacman as slowing down. The best explanations I could come up with are

1. I used to be more zealous about updating and did it frequently enough that the database rarely dropped out of the cache
2. Some other program I use regurlarly (Firefox?, Emby?) has started accessing more files and pushing the database out of the cache faster
3. I switched to using yaourt which runs "pacman -Qdt" before its first sudo invocation (part of it looking for orphaned packages). This exacerbates the problem a little because I have to wait for the database to be read from disk, then enter my password, and then wait for pacman to update the package database. When I just use "sudo pacman -Syu", the password prompt is first and then the two waits are combined. However, I don't remember always noticing this pause with yaourt.

but based on apg's modification maybe there was a real slow down.

Offline

#43 2018-01-26 12:35:00

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

wsha wrote:

... frequently enough that the database rarely dropped out of the cache...
... pushing the database out of the cache faster

For the Nth time (I stopped counting a while back) there is no singular database to be cached or not.  If there was, none of the issues discussed in this thread would exist.  Which of course you would know if you actually read the thread before bumping it.  So I gather you probably wont read this either, so I'm not sure why I'm bothering ... and I hope you don't expect anyone else to bother with a thoughtful reply either.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#44 2018-01-26 13:36:58

apg
Developer
Registered: 2012-11-10
Posts: 211

Re: [SOLVED] Pacman -Q can take a long time to execute

Trilby wrote:

For the Nth time (I stopped counting a while back) there is no singular database to be cached or not.  If there was, none of the issues discussed in this thread would exist.  Which of course you would know if you actually read the thread before bumping it.  So I gather you probably wont read this either, so I'm not sure why I'm bothering ... and I hope you don't expect anyone else to bother with a thoughtful reply either.

This is really unnecessary pedantry.  There is *a* local database that consists of multiple files which can be individually cached and subsequently pushed out of the cache exactly as described.  The fact that there are multiple files is irrelevant to the theory that wsha's recent usage patterns have started pushing them out of the cache.

wsha wrote:

Wow, apg, that modified version does seem to have helped. My access time has dropped from 20 seconds to 4-5 seconds. How does that modification make a difference?

The modification to pacman-optimize puts the files on the disk in sorted order, which is generally the order in which pacman will read them.

Offline

#45 2018-01-26 13:55:02

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

It's not pedantry.  The entire problem in this thread was the misunderstanding that different pacman operations access different subsets of files.  The view of a singular database being in a binary state of either recently used and cached or not recently used and not cached is what caused all the confusion here.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#46 2018-01-26 13:58:43

wsha
Member
Registered: 2015-07-22
Posts: 19

Re: [SOLVED] Pacman -Q can take a long time to execute

apg, how does that change that I quote result in a sorted order? I thought the -C option was equivalent to calling cd before tar, so the two versions look functionally equivalent to me. Is there a subtle difference between -C and cd first?

Trilby, I know that the database is not a single file. If you look at my first post to the thread, you will see that I talk about using strace pacman to watch all of the different open and read syscalls. Even Xyne refers to these files as a database in his last post.

Edit: I just want to point out there appear to be two slow downs of pacman referred to in the thread. One is the trivial one of reading in the database from disk vs. reading from the cache. The other is that reading from disk without a cache on a fresh install vs. on an older one for which the files on disk have become fragmented. At 4-5 seconds, I might overlook the time to read the database from disk, but at 20 seconds it is definitely something that I notice.

Last edited by wsha (2018-01-26 14:05:32)

Offline

#47 2018-01-26 14:32:43

apg
Developer
Registered: 2012-11-10
Posts: 211

Re: [SOLVED] Pacman -Q can take a long time to execute

The glob in my version returns the database entries in sorted order.

Offline

#48 2018-01-29 00:38:30

wsha
Member
Registered: 2015-07-22
Posts: 19

Re: [SOLVED] Pacman -Q can take a long time to execute

apg wrote:

The glob in my version returns the database entries in sorted order.

Ah, I missed that "*". That explains the change from -C to cd in a subshell. I was going to ask if you would submit your modification to the pacman maintainers, but I see that pacman-optimize was removed from the repo with a comment about it not doing as much as people thought it did. Maybe if your change had been merged in it would have been more effective and less likely to be removed (though it probably still would be removed eventually with everyone moving to SSD's).

Offline

#49 2018-01-29 00:44:10

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] Pacman -Q can take a long time to execute

There is a separate pacman-contrib repo now.  apg know about this, so I'd guess it will appear upstream.

Offline

#50 2018-01-29 03:48:45

apg
Developer
Registered: 2012-11-10
Posts: 211

Re: [SOLVED] Pacman -Q can take a long time to execute

wsha wrote:

I was going to ask if you would submit your modification to the pacman maintainers, but I see that pacman-optimize was removed from the repo with a comment about it not doing as much as people thought it did. Maybe if your change had been merged in it would have been more effective and less likely to be removed (though it probably still would be removed eventually with everyone moving to SSD's).

I've wasted enough time trying to convince people that, even if it's not always effective, pacman-optimize can be a useful tool and not just a placebo.  I have little interest in pursuing it further, but if you'd like to try to get pacman-optimize adopted by the pacman-contrib project, you certainly have my blessing to include my change.

Offline

Board footer

Powered by FluxBB