Slow gpg signature check

ZenRen · 2023-09-20 11:58:33

I've always had thoughts around the length of time the "Checking package integrity" step when running pacman updates. Sometimes its nearly as long as the download, but its not a parallel process. I am really not well versed in the subject, all I know beyond that it is validating the package checksums. Everything from here on is just my understanding from sifting through the wiki for specifically pacman's process of integrity checking. I am potentially looking for a speed up, though probably minor, and I don't want to reduce security, but I would really just like to learn a bit more on the topic in general.

Edit: I didn't want to delete any of my original post, but I soon learned the "Checking package integrity" step is running gpg, not validating a checksum. I am not sure if it is actually a problem, but it is something I would like to know more about, and if it can be sped up, that would be great.

So, with my understanding, the INTEGRITY_CHECK line in /etc/makepkg.conf is what controls the algorithm. However, I'm not sure if this is just what would be used for something like an AUR package and the algorithm is just hard baked into pacman for the main repos. The default setting is sha256, I was curious what adding or changing that to BLAKE2 would do. First of all https://wiki.archlinux.org/title/PKGBUILD#Integrity states that "When multiple types are available, the strongest checksum is to be preferred: b2 over sha512, sha512 over sha384, sha384 over sha256, sha256 over sha224, sha224 over sha1, sha1 over md5, and md5 over ck.", then why doesn't the INTEGRITY_CHECK line just include all of them? Is it because it would also generate all of them, which would just be slow, or if a lesser checksum passes then that is all it needs to be valid? If I were to replace sha256 with b2 would basically ever package fail an integrity check because I am not sure I have ever seen a PKGBUILD with anything besides sha256 or MD5. I'm not sure how it works with precompiled packages, I know there is a sha256 checksum in the .BUILDINFO file that I am assuming is what the "Checking package integrity" step runs against.

Last edited by ZenRen (2023-09-20 17:30:38)

Scimmia · 2023-09-20 12:27:22

So many assumptions here, most of which are wrong.

First, realize that pacman and makepkg are two separate things. Everything in your second paragraph is incorrect. You say it's validating the package checksum, but in most cases, it's actually checking the gpg signature; similar, but not the same. In the cases where there is no sig, it does check the checksum against the one in the package database.

ZenRen · 2023-09-20 13:36:01

Yeah, but I'm fine with that, I figured I'd ask and clear it up.

I know pacman and makepkg are separate things, but I didn't know if they shared settings. Typically pacman would be run after makepkg is finished. I kind of expected that they didn't share settings, that's why I wondered if what I was looking for is "baked in" to pacman. If "checking package integrity" in the context of pacman means checking the gpg signature then that is understandable, however I just assumed that was done in the step before it "checking keys in the keyring".

I guess gnupg is just really slow, or maybe its the speed of the terminal? I wouldn't have thought it would take much horsepower to check a signature with what is in your keyring, which lead me to assume it was validating a checksum.

Trilby · 2023-09-20 14:55:56

ZenRen wrote:

I just assumed that was done in the step before it "checking keys in the keyring".

I'm pretty sure that step is just checking whether the needed keys are available. Then the next step uses those keys to verify package integrity.

I have never experienced this step taking any notable amount of time - certainly no where near the time needed to download packages. How long is it taking for you? This is completely an X-Y problem: you've observed slow integrity checking and rather than providing information about that actual problem, you pursued questions about what you thought may be the cause. If you want help with the actual problem, you'll need to provide information about the problem itself, beyond it feeling slow.

A good test would be to run the gpg command to check some signatures outside of the context of pacman and time this process. I don't know this command off the top of my head - I suspect others could chime in with it, or if needed we could hunt it down.

Additionally, if you want to troubleshoot this actual problem, you should edit your thread title to reflect the actual concern / question.

Also, one simple check would be `cat /proc/sys/kernel//random/entropy_avail` (especially during a slow signature checking). And related to this, is this system perhaps a remote server, or is this a local machine with a physical keyboard / mouse / etc (regular laptop / desktop system)?

Last edited by Trilby (2023-09-20 15:02:41)

ZenRen · 2023-09-20 17:26:37

Ok, I probably exaggerated on how long it takes. Maybe a little less time than the downloads, outside of the larger packages. Downloads have a metric of speed at least, and while nowhere near the speed of what my internet or drives are capable of, pretty fast, its a large amount of data compared to what gpg is handling. The time it takes is just definitely noticeable in larger updates. I have 3 machines to compare, all with wildly varying hardware. One desktop with am R9-7900X, a laptop with an i7-3740QM, and another laptop with an R7Pro-3700U, all local machines. I have no reference to how fast the desktop is, but it just did a 20 package update and I didn't even see the step go by. I updated the Ryzen laptop last night, about 250 packages, and I would guess it took around 15 or 20 seconds. The Intel laptop just did an update, also around 250 packages and was probably 10 seconds despite the fact that it should be several orders of magnitude slower. So maybe you are onto something, maybe the Ryzen laptop has a problem.

I would love to check the speed of gpg on each, so I would love to know how to do that, I'd do a few runs with different amounts of signatures to check.

loqs · 2023-09-20 17:52:47

I think the gpg command may be:

$ gpg --home=/etc/pacman.d/gnupg/ --verify $something.pkg.tar.zst.sig $something.pkg.tar.zst

Trilby · 2023-09-20 18:10:31

An example test:

$ pkg=/var/cache/pacman/pkg/linux-6.5.3.arch1-1-x86_64.pkg.tar.zst

$ time gpg --home=/etc/pacman.d/gnupg/ --verify $pkg.sig $pkg
...
real	0m 0.43s
user	0m 0.42s
sys	0m 0.00s

Obviously the time will vary on different hardware, but drastic deviations could indicate a problem.

ZenRen · 2023-09-21 00:07:07

I did a few runs of the single verification here are the results:

$ time gpg --home=/etc/pacman.d/gnupg/ --verify /var/cache/pacman/pkg/linux-zen-6.5.3.zen1-1-x86_64.pkg.tar.zst.sig /var/cache/pacman/pkg/linux-zen-6.5.3.zen1-1-x86_64.pkg.tar.zst

R9-7900X     0.00s user 0.00s system 0.003s total
R7Pro-3700U  0.09s user 0.09s system 0.199s total    <== this is the one I thought was slow
i7-3740QM    0.47s user 0.02s system 0.496s total    Good try little buddy

Then I spent a couple hours writing a script to test the entire package cache... Here is the script https://gist.github.com/brendenhoffman/ … 7317864a1d. Wish I could do something about the overhead of the shell and updating the terminal, but I'd think this would be similar to what pacman runs in the "Checking package integrity" step. One other thing I'm unsure of is the speed of gpg when a key is good vs bad. Arch on my desktop is a few days old, all of the keys are good, that is not the case for the laptops, the Ryzen laptop is the oldest and most bloated. so it had the most bad signatures, it was also running Zoom. That being said, the desktop also had the fewest packages in the cache, so I did a few runs of 1400 iterations on each pc.

$ time ./gpgbench.sh 1400

R9-7900X      4.96s user  2.19s system  7.181s total
R7Pro-3700U  24.15s user 13.83s system 38.802s total    <== Definitely should not be the slowest!
i7-3740QM    27.89s user  4.86s system 32.842s total

And I ran cat /proc/sys/kernel/random/entropy_avail several times while running. They all reported 256, no changes under any scenario. That was a fun little experiment though.

Edit: Also, all of my machines are on the 6.5 zen kernel, if that wasn't obvious by my choice of package to verify, so they are all equal on that front. Later, after I finished my Zoom meeting, I ran the script on my Ryzen laptop again and only had gains of a few seconds. It is still several seconds slower on average than the old Intel laptop, for whatever that's worth.

Last edited by ZenRen (2023-09-21 02:21:48)

Arch Linux

#1 2023-09-20 11:58:33

Slow gpg signature check

#2 2023-09-20 12:27:22

Re: Slow gpg signature check

#3 2023-09-20 13:36:01

Re: Slow gpg signature check

#4 2023-09-20 14:55:56

Re: Slow gpg signature check

#5 2023-09-20 17:26:37

Re: Slow gpg signature check

#6 2023-09-20 17:52:47

Re: Slow gpg signature check

#7 2023-09-20 18:10:31

Re: Slow gpg signature check

#8 2023-09-21 00:07:07

Re: Slow gpg signature check

Board footer