You are not logged in.

#1 2022-09-19 01:25:14

Salkay
Member
Registered: 2014-05-22
Posts: 599

archlinux-keyring-wkd-sync.service constantly failing

archlinux-keyring-20220831-1 installed a new systemd service that seems to constantly fail.

# systemctl status archlinux-keyring-wkd-sync.service
× archlinux-keyring-wkd-sync.service - Refresh existing keys of archlinux-keyring
     Loaded: loaded (/usr/lib/systemd/system/archlinux-keyring-wkd-sync.service; static)
     Active: failed (Result: exit-code) since Mon 2022-09-19 07:19:31 AEST; 3h 57min ago
   Duration: 274ms
TriggeredBy: ● archlinux-keyring-wkd-sync.timer
    Process: 1768430 ExecStart=/usr/bin//archlinux-keyring-wkd-sync (code=exited, status=2)
   Main PID: 1768430 (code=exited, status=2)
        CPU: 247ms

Sep 19 07:19:31 hostname systemd[1]: Started Refresh existing keys of archlinux-keyring.
Sep 19 07:19:31 hostname archlinux-keyring-wkd-sync[1768430]: Skipping key 51588BCC4F03C4FAA8FAFC09887B16AB27243B9B with UID pacman@localhost...
Sep 19 07:19:31 hostname archlinux-keyring-wkd-sync[1768430]: Skipping key AB19265E5D7D20687D303246BA1DFB64FFF979E7 with UID allan@master-key.archlinux.org...
Sep 19 07:19:31 hostname archlinux-keyring-wkd-sync[1768430]: Skipping key DDB867B92AA789C165EEFA799B729B06A680C281 with UID bpiotrowski@master-key.archlinux.org...
Sep 19 07:19:31 hostname archlinux-keyring-wkd-sync[1768430]: Refreshing key 91FFE0700E80619CEB73235CA88E23E377514E00 with UID florian@master-key.archlinux.org...
Sep 19 07:19:31 hostname archlinux-keyring-wkd-sync[1768596]: gpg: error retrieving 'florian@master-key.archlinux.org' via WKD: Server indicated a failure
Sep 19 07:19:31 hostname archlinux-keyring-wkd-sync[1768596]: gpg: error reading key: Server indicated a failure
Sep 19 07:19:31 hostname systemd[1]: archlinux-keyring-wkd-sync.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 19 07:19:31 hostname systemd[1]: archlinux-keyring-wkd-sync.service: Failed with result 'exit-code'.

I also found a Reddit thread referring to this issue. Should this job be working, and/or should we just mask it?

Offline

#2 2022-09-19 06:37:01

seth
Member
Registered: 2012-09-03
Posts: 33,672

Re: archlinux-keyring-wkd-sync.service constantly failing

sudo gpg --homedir /etc/pacman.d/gnupg --search-keys florian@master-key.archlinux.org # yes, must be UID0
pacman-key --list-keys florian@master-key.archlinux.org

https://wiki.archlinux.org/title/Pacman … _keyserver

Online

#3 2022-09-19 09:43:21

Salkay
Member
Registered: 2014-05-22
Posts: 599

Re: archlinux-keyring-wkd-sync.service constantly failing

Thanks @seth.

$ sudo gpg --homedir /etc/pacman.d/gnupg --search-keys florian@master-key.archlinux.org
gpg: WARNING: unsafe permissions on homedir '/etc/pacman.d/gnupg'
gpg: data source: https://162.213.33.9:443
(1)	Florian Pritz (Arch Linux Master Key) <florian@master-key.archlinux.or
	  4096 bit RSA key A88E23E377514E00, created: 2015-12-17
Keys 1-1 of 1 for "florian@master-key.archlinux.org".  Enter number(s), N)ext, or Q)uit > 1
gpg: key A88E23E377514E00: 2 duplicate signatures removed
gpg: key A88E23E377514E00: "Florian Pritz (Arch Linux Master Key) <florian@master-key.archlinux.org>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
$ pacman-key --list-keys florian@master-key.archlinux.org
gpg: Note: trustdb not writable
pub   rsa4096 2015-12-17 [SC]
      91FFE0700E80619CEB73235CA88E23E377514E00
uid           [  full  ] Florian Pritz (Arch Linux Master Key) <florian@master-key.archlinux.org>
sub   rsa4096 2015-12-17 [E]

I'm not entirely sure what I'm doing here. Was this just to confirm that the key existed and was accessible on the server?

Also, it looks like I'm already using the Ubuntu keyserver.

$ cat /etc/pacman.d/gnupg/gpg.conf
no-greeting
no-permission-warning
lock-never
keyserver hkps://keyserver.ubuntu.com
keyserver-options timeout=10

Offline

#4 2022-09-19 12:30:48

seth
Member
Registered: 2012-09-03
Posts: 33,672

Re: archlinux-keyring-wkd-sync.service constantly failing

Was this just to confirm that the key existed and was accessible on the server?

Yes, doesn't look there's any problem.
Does running

sudo archlinux-keyring-wkd-sync

directly cause any problems?

Online

#5 2022-09-19 20:25:41

NiceGuy
Member
Registered: 2018-02-19
Posts: 50

Re: archlinux-keyring-wkd-sync.service constantly failing

At first, with the introduction of the additional service and timer units I also noticed the errors and in the end I decided to masked them in the meantime after experimentation made no difference.

@seth: Do you think it's just related to WKD servers? Nothing else was changed since I tried it, now it just behaves as intended.

Also there is a minor typo in the archlinux-keyring-wkd-sync.service in ConditionFileIsExecutable and ExecStart.  The path: /usr/bin//archlinux-keyring-wkd-sync is happily executed and the typo makes no difference here, wondered why.


What's odd, there is no different output of archlinux-keyring-wkd-sync no matter how often it is invoked via systemd timer or manually. Does this seem right to you?
I had the impression, after refreshing certain keys, that those keys would be skipped and the process of keyring synchronization would end quicker.

Last edited by NiceGuy (2022-09-19 20:26:42)

Offline

#6 2022-09-19 20:40:07

seth
Member
Registered: 2012-09-03
Posts: 33,672

Re: archlinux-keyring-wkd-sync.service constantly failing

there is no different output of archlinux-keyring-wkd-sync no matter how often it is invoked

It seems to unconditionally refresh all keys and only skip double entries (same key, different ID)

My current theory for Salkay's situation would be that the timer hits before the network is up and that causes the error w/ a bogus status - nothing in #3 suggests any problem w/ the key or keyserver or Salkay's configuration.

Online

#7 2022-09-20 00:49:15

Salkay
Member
Registered: 2014-05-22
Posts: 599

Re: archlinux-keyring-wkd-sync.service constantly failing

Does running ... directly cause any problems?

@seth Hm, that worked fine when I just tested. I then tried to restart archlinux-keyring-wkd-sync.service, and this started fine now (as per journalctl), but it did fail partway with

Sep 19 23:54:43 hostname archlinux-keyring-wkd-sync[1953092]: Refreshing key 601F20F1D1BBBF4A78CF5B6DF6B1610B3ECDBC9F with UID crerar@archlinux.org...
Sep 19 23:54:49 hostname archlinux-keyring-wkd-sync[1953361]: gpg: error retrieving 'crerar@archlinux.org' via WKD: End of file
Sep 19 23:54:49 hostname archlinux-keyring-wkd-sync[1953361]: gpg: error reading key: End of file
Sep 19 23:54:49 hostname systemd[1]: archlinux-keyring-wkd-sync.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 19 23:54:49 hostname systemd[1]: archlinux-keyring-wkd-sync.service: Failed with result 'exit-code'.

I wonder if that was just a random network error. I suspect you are correct, and it's a network issue that is causing most of the errors. I do use a VPN, so perhaps it just takes a little while to establish network, which sometimes causes issues for the service.

EDIT: I also see this error on my server that is always connected to the network, so I suspect that any temporary issue with connectivity to the server causes this service to fail.

EDIT2: This seems to occur every few days, even on a stable connection. I just run sudo systemctl restart archlinux-keyring-wkd-sync.service, which seems to fix it.

Last edited by Salkay (2022-10-22 05:32:24)

Offline

#8 2022-10-25 19:17:41

MrDuck
Member
Registered: 2022-10-25
Posts: 2

Re: archlinux-keyring-wkd-sync.service constantly failing

Hope this is not necro-bumping but title seemed appropriate and may be a "version-agnostic" potential solution ...

If the /usr/bin/gpg command fails, on line 58 of the service script, the script will terminate with an error and so the service fails.

If we change line 58 of /usr/bin/archlinux-keyring-wkd-sync from:

"${gpg_locate_external[@]}" "${fpr_email[1]}"

to:

"${gpg_locate_external[@]}" "${fpr_email[1]}" || true

... then the script will continue to process the remaining keys without error.

Perhaps the script should be patched to do this?

Offline

#9 2022-10-25 19:46:59

seth
Member
Registered: 2012-09-03
Posts: 33,672

Re: archlinux-keyring-wkd-sync.service constantly failing

This would be better discussed at https://gitlab.archlinux.org/archlinux/ … x-keyring/

Online

#10 2022-10-25 20:16:15

MrDuck
Member
Registered: 2022-10-25
Posts: 2

Re: archlinux-keyring-wkd-sync.service constantly failing

Offline

#11 2022-10-25 23:25:33

Salkay
Member
Registered: 2014-05-22
Posts: 599

Re: archlinux-keyring-wkd-sync.service constantly failing

Thanks for the detective work @MrDuck and thanks for filing the issue. Good to know, and I can always patch it if it's not changed upstream.

Offline

#12 2022-11-22 08:25:53

drankinatty
Member
From: Nacogdoches, Texas
Registered: 2009-04-24
Posts: 44
Website

Re: archlinux-keyring-wkd-sync.service constantly failing

I am hit by this very problem after doing a $ sudo pacman -Syu. I have journal errors that are 10+ lines long for what looks like every key in the keyring, e.g.

Nov 22 02:20:01 valkyrie archlinux-keyring-wkd-sync[12851]: Refreshing key 64B13F7117D6E07D661BBCE0FE763A64F5E54FD6 with UID kpcyrd@archlinux.org...
Nov 22 02:20:16 valkyrie archlinux-keyring-wkd-sync[18893]: gpg: error retrieving 'kpcyrd@archlinux.org' via WKD: Connection timed out
Nov 22 02:20:16 valkyrie archlinux-keyring-wkd-sync[18893]: gpg: error reading key: Connection timed out
Nov 22 02:20:16 valkyrie archlinux-keyring-wkd-sync[12851]: Skipping key A2FF3A36AAA56654109064AB19802F8B0D70FC30 with UID jan.steffens@gmail.com...
Nov 22 02:20:16 valkyrie archlinux-keyring-wkd-sync[12851]: Skipping key A2FF3A36AAA56654109064AB19802F8B0D70FC30 with UID jan.steffens@ltnglobal.com...
Nov 22 02:20:16 valkyrie archlinux-keyring-wkd-sync[12851]: Refreshing key A2FF3A36AAA56654109064AB19802F8B0D70FC30 with UID heftig@archlinux.org...
Nov 22 02:20:31 valkyrie archlinux-keyring-wkd-sync[18896]: gpg: error retrieving 'heftig@archlinux.org' via WKD: Connection timed out
Nov 22 02:20:31 valkyrie archlinux-keyring-wkd-sync[18896]: gpg: error reading key: Connection timed out
Nov 22 02:20:31 valkyrie archlinux-keyring-wkd-sync[12851]: Skipping key 05C7775A9E8B977407FE08E69D4C5AA15426DA0A with UID frederik.schwan@linux.com...
Nov 22 02:20:31 valkyrie archlinux-keyring-wkd-sync[12851]: Skipping key 05C7775A9E8B977407FE08E69D4C5AA15426DA0A with UID frederik@schwan.it...
Nov 22 02:20:31 valkyrie archlinux-keyring-wkd-sync[12851]: Skipping key 05C7775A9E8B977407FE08E69D4C5AA15426DA0A with UID frederik@tty42.de...

The error messages should output the IP for the failing refresh so we can confirm it isn't part of a block that is in iptables. With many of the Arch servers spread all over the world, and part of IP blocks that in the past have been suspect within RIPE or other bodies, that would be helpful.

Has the fix been incorporated upstream?  The update went fine, the journal is just getting filled with the "Skipping key" errors.

Last edited by drankinatty (2022-11-22 08:35:00)


David C. Rankin, J.D.,P.E.

Offline

#13 2022-11-22 09:18:46

Salkay
Member
Registered: 2014-05-22
Posts: 599

Re: archlinux-keyring-wkd-sync.service constantly failing

It's "fixed" as per the linked issue, but it still fails for me every couple of days. I actually created an account to comment on this issue and mention the failing, but my comment was ignored.

What does this service actually do? Can we just mask it?

Offline

#14 2022-11-22 11:05:47

seth
Member
Registered: 2012-09-03
Posts: 33,672

Re: archlinux-keyring-wkd-sync.service constantly failing

systemctl show archlinux-keyring-wkd-sync.service wrote:

Description=Refresh existing keys of archlinux-keyring

It's to combat all the "Q: helps, update failed!!! / A: pacman -Sy archlinux-keyring; pacman -Syu" situations.
You're better off w/ the script "failing" on individual keys (typically because of local flaky network) and have them hopefully just sanitized than running into the keyring errors whenever you occasionally update. But you can mask it and then just remember to pre-update the keyring when you get related failures.

Online

#15 2022-11-22 11:32:29

Salkay
Member
Registered: 2014-05-22
Posts: 599

Re: archlinux-keyring-wkd-sync.service constantly failing

Thanks @seth, that makes sense.

Offline

Board footer

Powered by FluxBB