howto test updates before they are deployed to a fleet of machines?

cdwijs · 2025-07-14 16:29:41

I would like to be able to update a fleet of machines running Arch Linux. These machines will be deployed at remote sites. The people at those sites are not tech-savvy (enough) to fix problems.

I would like to update them automatically, but I would like to test the update before it's rolled out to all the machines.

Is there any way to do that?

cryptearth · 2025-07-14 16:40:45

maybe not use arch but a different more stable distro like debian or suse - and change mirrors to a server controlled by you rather the official repos
also: don't grant users admin privileges if they can't handle them
also an option: host a powerful terminal server and provide dumb terminal clients with a minimal set of packages

tldr: there several options - and the most appropriate answer will likely be "it depends (on several factors)"

schard · 2025-07-14 17:27:13

You should really consider using something other than Arch Linux for that scenario.
E.g. Yocto with automated updates via Mender.

That being said, in my previous company we also used Arch Linux administered remotely for various reasons.
However, we did not automate the updates, but rolled them in batches on the remote machines.
So if there was an issue, we would only have to fix it on one batch at a time.
Usually we started with a batch that was physically close to us, so that we could reach it quickly if a worst-case scenario occurred and we lost remote access.
We were quite successful with that and had a very low failure rate.

Edit: Mod note: Not an installation issue. Moving to System Administration.

Last edited by schard (2025-07-14 17:27:44)

herOldMan · 2025-07-14 22:22:35

Hi,

I use Arch as a base hypervisor on over three dozen commercial machines (8 years now). The organization is an analytical laboratory and the system configurations vary widely. I have used multiple other distros and I have found Arch preferable. I have had problems, but this forum provided me with timely answers. The resources of the Arch forum and the wiki are my primary reason for using Arch. Arch's granularity is another.

My approach:

I visit this forum daily, specifically the applications & desktop, networking and server. system admin, and pacman upgrade issues sub-forums.

I always do my upgrades on an isolated and representative set of machines before I go live in production.

Last edited by herOldMan (2025-07-14 22:23:37)

cdwijs · 2025-07-15 20:24:07

I was thinking along the lines of updating one machine at the office, and test it. If everything is OK, I would like the fleet outside to become identical to the machine at the office, meaning the versions of the packages in the fleet should be the same as the machine in the office.

Is it possible to gather a list of version numbers from the machine in the office, send that list to the fleet somehow, and let the fleet then update to exactly those versions from the official mirrors? I would like to avoid the situation that the entire fleet downloads the updates from my office, as that's will probably overwhelm that internet connection.

Is this a valid approach, or is this a very bad(tm) idea?

cryptearth · 2025-07-16 04:13:58

as far as I'm aware none of the package managers offer downloading specific versions - and I don't talk about AUR or flat/snap but the real distribution ones like APT on debian, zypper on suse or DNF on gentoo
arch, however, offers the archive repo which are daily snapshots - so instead of specific versions you can sync up to a specific time/date - but I'm not sure about how to distribute that
arch, as a rolling release with packages update sometimes multiple times per day, might not be a good choice overall
as for how to lower the load on a server: you could use caching mirrors so only a few downloads are from some master repo hosted on a server controlled by you to spread the load

Whoracle · 2025-07-16 09:55:41

as far as I'm aware none of the package managers offer downloading specific versions - and I don't talk about AUR or flat/snap but the real distribution ones like APT on debian, zypper on suse or DNF on gentoo

APT does. zypper does, too, and on arch can do it via downgrade PKGNAME=VERSION. If you install downgrade, that is. Or via the ALA.

But for the topic at hand, I'd set up a server as a full mirror, sync that against upstream every TIMEINTERVAL, have a few servers use that repo for testing, and have a 2nd mirror that sync off the first as the main repo for all other servers, and sync the 2nd one once I tested the upgrades via the first. You could even use the same machine for both repos. That way you could even upgrade the main fleet via a cronjob, but I'd rather use something like ansible or puppet for that, which IMHO you should use for any fleet deployment either way.

Last edited by Whoracle (2025-07-16 11:04:56)

cryptearth · 2025-07-16 11:09:12

zypper requires that the requested version is available on the mirror and downgrade is just a helper script providing a specific tarball - so, tldr: no, neither zypper nor pacman can install specific versions - they only can install given packages which have to be provided upfront in the requested version
as for apt - idc, I don't use debian or any of its forks

Whoracle · 2025-07-16 11:32:37

zypper requires that the requested version is available on the mirror

Yes, your point being? Any version (explicitly requested or no) needs to be either in a repo or on-disk. Where else would a package come from?

Point being: with zypper I can pass a package with a version, and as long as it's available, zypper will install said version. As will apt. pacman, on the other hand, does not even provide that functionality, because by design there's not multiple versions of the same package name in pacman-compatible repos. So "as far as I'm aware none of the package managers offer downloading specific versions" is false. pacman doesn't (with good reason, given its design).

Now, if one were to split hairs then the argument would be that neither apt nor zypper are actually package managers, but wrappers around dpkg and rpm/dnf, respectively, and "just download packages to then be installed with the actual package manager", and that would be true, but IMHO needlessly complicated for the purposes of this discussion. But looking at it that way, pacman is actually the only one (from the mentioned package managers, at least) that does downloading and installation. But again: What does that add to the discussion at hand?

Last edited by Whoracle (2025-07-16 11:36:46)

Arch Linux

#1 2025-07-14 16:29:41

howto test updates before they are deployed to a fleet of machines?

#2 2025-07-14 16:40:45

Re: howto test updates before they are deployed to a fleet of machines?

#3 2025-07-14 17:27:13

Re: howto test updates before they are deployed to a fleet of machines?

#4 2025-07-14 22:22:35

Re: howto test updates before they are deployed to a fleet of machines?

#5 2025-07-15 20:24:07

Re: howto test updates before they are deployed to a fleet of machines?

#6 2025-07-16 04:13:58

Re: howto test updates before they are deployed to a fleet of machines?

#7 2025-07-16 09:55:41

Re: howto test updates before they are deployed to a fleet of machines?

#8 2025-07-16 11:09:12

Re: howto test updates before they are deployed to a fleet of machines?

#9 2025-07-16 11:32:37

Re: howto test updates before they are deployed to a fleet of machines?

Board footer