healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

triam · Yesterday 07:23:27

I once broke my system by upgrading and rebooting at just the wrong moment during a mirror sync. When the top mirror is out of sync, pacman can end up installing a mix of old and new packages, and it is easy to miss. After seeing similar reports following big upgrades on the CachyOS forum, I wrote a script that checks the Arch/CachyOS mirror status APIs and comments out out-of-sync mirrors with a #UH# marker (reversible with "hm --clean"). HTTPS mirrors only, which keeps the code simpler. It works on vanilla Arch too.

The project has since matured considerably (yes, vibe-coded!). It now includes an optional, security-hardened daemon that detects package manager activity and checks/fixes mirrors before a transaction. Tested with pacman, paru, Shelly and cachy-update. It runs as root, uses fanotify and a sandboxed systemd unit. By default it blocks the package manager when no healthy mirror is found; starting it again within 60 s lets it through.

No daemon needed, though: running "hm" manually typically takes under a second on my connection ("hm --dry-run" shows what would change).

Install:
git clone https://github.com/mtriam/healthy-mirrors.git && cd healthy-mirrors && makepkg -si && sudo hm-setup --defaults
(the last step enables the optional daemon; skip it if you only want to run hm by hand)

Please review the code before running it as root. Feedback, bug reports and suggestions are very welcome.

https://github.com/mtriam/healthy-mirrors

cryptearth · Yesterday 07:34:20

https://bbs.archlinux.org/misc.php?action=rules

These forums are for Arch Linux x86_64 ONLY.
Not Artix, or Apricity, or Manjaro, or any of the "easy Arch installers", nor Arch-ARM; nothing other than vanilla 64-bit Arch Linux.

by the time you're registered here you should know

also: instead of such tool you should rather recheck your mirrorlist

... although i do agree that "just using next mirror without checking its state" leading up to version mis-matches is an inherent problem of how pacman handles mirror failover

Last edited by cryptearth (Yesterday 07:36:23)

seth · Yesterday 08:18:45

https://archlinux.org/packages/extra/any/reflector/ didn't work for you which is why you *had* to use your new AI toy?

triam · Yesterday 09:26:07

@cryptearth: agreed on the failover problem, that is what this guards against.

@seth: I compared: "reflector --protocol https --delay 1" returns exactly the same healthy HTTPS mirrors that hm keeps. Today's snapshot: less than half of the HTTPS mirrors are good (207 of 471). The difference is timing, not criteria: reflector with --sort rate before every upgrade means a speed test each time (and needless load on the mirrors if everyone did it), while the default timer only runs weekly. So rank with rate-mirrors/reflector now and then, and sanitize right before the transaction with hm, or let hm-daemon do it automatically.

Vibe-coded, so please point out any errors. Tested on a vanilla Arch as well.

Lone_Wolf · Yesterday 10:25:19

The failover mechanism is not perfect , but in which circumstances / what pacman invocation did this occur ?

Moderator Note
It's questionable whether such breakage due to an out-of-sync top mirror has happened on vanilla archlinux systems.

Please provide proof this does affect archlinux systems.

triam · Yesterday 12:04:46

It happened over a year ago on CachyOS with a plain "sudo pacman -Syu". I have no documented case on vanilla Arch.
What I can show on Arch is the mechanism. Pacman never checks whether a mirror is in sync. "pacman -Sy --debug" shows core.db and extra.db requested separately, so with failover they can in principle come from mirrors in different states. Consistency then rests on declared dependencies, and some are unversioned: "pacman -Si qt6-declarative" depends on a plain "qt6-base" and qt6-base provides nothing, so a mismatched pair would pass the resolver.

Trilby · Yesterday 13:25:18

Wow, this is just what I needed. I have a half-dozen lines of simple shell script that completes the same task reliably and without fail for years - but the opportunity to replace that with thousands of lines of AI slop which includes a daemon and a system service is surely an advancement in efficiency for my workflow and will help me outpace the deliverables of my colleagues.

Who wouldn't want an always-on always-running daemon in the background executing mysterious code. That just adds to the adventure of running arch linux: leave the doors wide open as a trap for malicious actors: gotta catch them all!

Last edited by Trilby (Yesterday 13:26:50)

seth · Yesterday 13:44:21

So rank with rate-mirrors/reflector now and then, and sanitize right before the transaction with hm, or let hm-daemon do it automatically.

Just to be clear, the idea is to simply correlate the local (ranked) mirror list w/ https://archlinux.org/mirrorlist/?count … _status=on right before the update?
Or maybe even rather just check the /lastsync of the ranked mirrors to filter out/warn about old ones?

Fwwi

#!/bin/bash
epoch=$(date +%s)
rank=1
while read mirror; do
    lastsync="$(curl -sL ${mirror}/lastsync 2>/dev/null)"
    ((epoch - lastsync < 3600)) || echo "$mirror - rank $rank not synced in one hour!"
    ((++rank))
done < <(sed '/^Server = /!d; s%^Server = \(.*\)$repo.*%\1%g' /etc/pacman.d/mirrorlist)

Last edited by seth (Yesterday 14:07:31)

cryptearth · Yesterday 14:06:53

i gave it another thought - and have to revise my reply: although timing differences can lead to de-sync between master and mirrors and between mirrors, the issue as described here should always result in a hard error and leaving the system untouched

why: a package's file includes several meta data, like its version
so if one sync with mirror A but for some reason failover to mirror B either mirror B can serve the requested file - or not, resulting in another failover

so

pacman can end up installing a mix of old and new packages

should not be possible even IF the failover happens right at the beginning at downloading the .db files

IF you end up with core.db from mirror A and extra.db from mirror B - which can happen - but they're both not in sync with eachother - this would lead to an error when pacman is determine version conflicts, which happens local

as you mentioned

happened on CachyOS

i may point again to the rules - as it sounds like some issue specific to cachy and whatever they did to let this happen

anyone feel free to prove me wrong - but i just don't see how the issue could even happen as described

Last edited by cryptearth (Yesterday 14:07:56)

triam · Yesterday 15:31:47

@seth: yes, the first one: hm correlates your ranked list with the official status data right before the transaction and comments out mirrors that are inactive, not fully synced or lagging by an hour or more. Your /lastsync check is a fair alternative with one advantage: it is current. hm trusts the API snapshot and doesn't check its age (when I fetched it just now it was 20 min old).

Cost today: one request per run to archlinux.org (the status JSON, ~480 KB; the server can send ~62 KB gzipped, which hm doesn't request yet), nothing to the mirrors themselves; with the daemon at most once per 60 s. I'm thinking about an optional /lastsync check of the first few active mirrors: a handful of tiny requests to those mirrors per run, off by default. If you only want a warning, your script does the job.

@cryptearth: agreed, as long as packages declare the versions they need, the resolver stops the transaction. The problem is only where they don't (like in the qt6 example above), then nothing catches a mismatched pair. I haven't seen that happen via mirrors on Arch, so I agree it should be rare.

@Trilby: fair concern about the daemon, which is why it's optional: hm alone is script you run yourself, and the daemon only automates that. Your own script for the same task suggests the problem is real.

cryptearth · Yesterday 21:24:09

triam wrote:

@cryptearth: agreed, as long as packages declare the versions they need, the resolver stops the transaction. The problem is only where they don't (like in the qt6 example above), then nothing catches a mismatched pair. I haven't seen that happen via mirrors on Arch, so I agree it should be rare.

i somewhat smell a misunderstanding about package versions and relying on a specfic api:

when a dependend package only declares a dependency but not a specific version then this means the dependend only relies on the stable public api the dependency provides not matter the specific version
so, if some package declares: "i need qt6-base" that specifically means "i just rely on ANY version of qt6 - no matter which" and hence, correctly, doesn't declare a specific version
but: that's the burden of the developer (and packager) to make sure that this doesn't break for the livetime of the dependency - in case of your example that means: the dependend just need functions from qt6-base that were already there with the initial 6.0 release and have not changed all the way to the most recent 6.12 version - if this condition is satisfied then it's correct to just reference qt6-base without any version
only if the dependend at some point start to rely on some feature that was not part of the initial 6.0 release but was added at some later version then a specific minimum version has to be declared
if, on top of the above, the dependend relies on some function that was removed with a more recent patch only then a specific version has to be declared

which brings me full circle back around to your issue as reported: the issue you encountered was very likely some slip up of the maintainer of some dependend package not correctly specifying thier actual dependencies - but this is not a flaw of pacman of the mirror failover but a packaging error - one even your tool will not help to prevent - which, as hard as it sounds, makes it completely obsolete from the very moment your wrote it
and i have to agree with others here: as the specific issue should not happen when both the dependend and the dependency are done correctly - by providing either a stable api or a version specific one and the dependend correctly specify what it requires - then only few will think about relying code which the pacman eco-system does itself already and which doesn't protect an packaging error

Arch Linux

#1 Yesterday 07:23:27

healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#2 Yesterday 07:34:20

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#3 Yesterday 08:18:45

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#4 Yesterday 09:26:07

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#5 Yesterday 10:25:19

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#6 Yesterday 12:04:46

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#7 Yesterday 13:25:18

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#8 Yesterday 13:44:21

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#9 Yesterday 14:06:53

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#10 Yesterday 15:31:47

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

#11 Yesterday 21:24:09

Re: healthy-mirrors – keeps out-of-sync mirrors from breaking your upgrade

Board footer