Arch Linux 'stable' repositories for servers?

timg · 2016-03-30 11:24:49

Hi Everyone,

I have been having this thought in my head for the last couple of months, and I feel like I really need to go and finally put pen to paper on this idea, so I can finally get some sleep!

A long time user of Arch Linux, I have been thinking about how I can get it on to servers. This has recently become a sore point with me, because at work, we mainly deploy RHEL servers (which, don't get me wrong, I love), but one of our partners are hell bent on only deploying their software on Debian-based servers.

This got me thinking, what would a stable, reliable "Arch Linux for Servers" look like?

There are already two very big models here that are worth a quick overview.

Fedora + Red Hat Enterprise Linux / CentOS
Debian + Ubuntu LTS

Essentially, the LTS and Enterprise versions of the above distro's take a snapshot of the latest release of the distro, and put them through a more thorough QA process (I understand this is an oversimplification of the process)

So what if we did the same thing for Arch Linux?

If we did a snapshot of the base repository, and a selection of the extra repository packages while keeping a primary focus on server-related packages, such as databases, programming languages (gcc, llvm, go, swift, php, ruby, python etc), moving versions of primary packages such as core libraries to the latest stable / "LTS" release upstream, then Arch Linux could essentially create it's own -lts branch.

My thought is that we would snapshot Arch say every 12 months, performing the process described above, where we ensure we are not getting "bleeding edge" packages, but rather "battle tested" packages. This snapshot would be versioned (say version 1, or version 2016).

We would then ensure that the snapshots are supported for 24-36 months.

This would mean quite a bit of work, such as ensuring that packages are kept up to date. This work would become harder the further a release comes to it's end of life.

Patches for packages would be released only if they fix bugs (crashes, unexpected behaviour or security issues). A version bump for a package would only be allowed if it did not add any new features, only fixed a bug. Everything else would require being patched, with a bump in the package release number, with a patch being provided for the reason of the package bump.

As part of this process, we would be able to ensure that:

Kernels would only run LTS versions, and the kernel version would not change to a new major/minor version unexpectedly
Application Binary Interface would not change on a particular release
Versions of scripting languages would not have unexpected version bumps (no python 2 --> python 3, etc)
Databases (postgres, mysql) would be pinned to the latest minor release (ie, MySQL won't move from 5.7.x, etc)

There is no reason why repositories could be provided if people wanted, say, a newer version of Go, PHP or postgres running on their server then what is supported in the official "snapshot" repository.

I guess my questions are:

1. Would anyone be interested in a project like this (or would you be unlikely to move from your current Debian/CentOS boxes).
2. Would anyone be interested in giving a helping hand in a project like this (tracking CVE's, helping with build scripts etc).

Allan · 2016-03-30 12:32:36

FYI - an Arch Server project was around many years ago. Did not gain traction to not enough man power and died...

timg · 2016-03-30 12:40:19

Yes, I realise this. This is the reason for asking the questions, and putting some of my thoughts down onto paper.

tho068 · 2016-04-01 16:36:53

Would not mind having a more stable Arch for servers. However, I don't really see it happening. It's a lot of work maintaining a whole set of packages, patching and all that.

I do use Arch on my server. Works fine most of the time.

TheChickenMan · 2016-04-01 18:21:10

I also think it would be a whole lot of work (and commitment for a long time of support) for a rather small user base that might want to use it. There are other distributions which work great as servers if you need package stability, like CentOS. I do have a couple of internal servers which do not have internet access running Arch. I use a snapshot of the repositories to install "new" packages if they are needed. It works great but I wouldn't trust that to be secure online forever.

kokoko3k · 2016-04-01 19:05:16

TheChickenMan wrote:

I use a snapshot of the repositories to install "new" packages if they are needed.

Why don't you use Arch rollback machine for that?

Last edited by kokoko3k (2016-04-01 19:06:02)

TheChickenMan · 2016-04-01 19:35:45

kokoko3k wrote:

TheChickenMan wrote:
I use a snapshot of the repositories to install "new" packages if they are needed.
Why don't you use Arch rollback machine for that?

Because these computers do not have internet access and connect only to an offline LAN.

2ManyDogs · 2016-04-01 20:34:00

TheChickenMan wrote:

I also think it would be a whole lot of work (and commitment for a long time of support) for a rather small user base that might want to use it. There are other distributions which work great as servers if you need package stability, like CentOS.

Exactly this. I don't understand why people want to make Arch into something it's not, just to run it on a server. There are plenty of other distros that work well on servers and have people already working to maintain them.

jryan · 2016-04-12 20:35:46

2ManyDogs wrote:

TheChickenMan wrote:
I also think it would be a whole lot of work (and commitment for a long time of support) for a rather small user base that might want to use it. There are other distributions which work great as servers if you need package stability, like CentOS.
Exactly this. I don't understand why people want to make Arch into something it's not, just to run it on a server. There are plenty of other distros that work well on servers and have people already working to maintain them.

I could see the reasoning. I know that I'd rather use a stable Arch on my servers so I have full control, and know how to do things; since other distros do things in different ways.

brebs · 2016-04-12 20:57:09

timg wrote:

Patches for packages would be released only if they fix bugs (crashes, unexpected behaviour or security issues). A version bump for a package would only be allowed if it did not add any new features, only fixed a bug. Everything else would require being patched, with a bump in the package release number, with a patch being provided for the reason of the package bump.

LOL, that's a dream-world. Here is what a typical version bump of a typical package contains:

* 3 critical bug-fixes, making it a "required" update for security reasons. One of which makes an unmentioned, subtle API/ABI change which will subtlely break some other dependant library/app which hasn't been recompiled.
* 2 new, experimental API functions which are bug-ridden, one of which is spaghetti-code-interweaved with a critical bug-fix.
* 1.5 completely new bugs, introduced as part of the critical bug-fixes.

Warning: The above may contain cynicism

ayekat · 2016-04-12 21:13:55

jryan wrote:

since other distros do things in different ways.

They usually do things in a different way because they have that whole "stable releases" thing going on.

You can't just start putting software packages into "freezed" state on a regular basis, keep backporting bug fixes for old software versions, test that the random selection of packages with a bunch of patches strapped onto them works flawlessly together, and recruit tons of people to build up an infrastructure for maintaining all that workflow (because that's the only correct way to do stable releases), while at the same time claim to be a "vanilla and KISS, yay!" distribution.

And no, having simply an artificial delay "with additional testing" until packages reach the "stable" stage, but otherwise keep that stage rolling, (like some Arch derivatives do) doesn't solve anything. I think I'm not wrong to say that roughly 90% of Arch Linux maintainance is not caused by mere breakage/bugs, but by continuous API/ABI/configuration changes during software upgrades. These problems would still hit "stable" - just a little later.

timg · 2016-04-17 07:38:55

brebs wrote:

Here is what a typical version bump of a typical package contains:
* 3 critical bug-fixes, making it a "required" update for security reasons. One of which makes an unmentioned, subtle API/ABI change which will subtlely break some other dependant library/app which hasn't been recompiled.
* 2 new, experimental API functions which are bug-ridden, one of which is spaghetti-code-interweaved with a critical bug-fix.
* 1.5 completely new bugs, introduced as part of the critical bug-fixes.

Well, then skip the version bumps all together and only allow patches that fix bugs and/or security issues.

ayekat wrote:

And no, having simply an artificial delay "with additional testing" until packages reach the "stable" stage, but otherwise keep that stage rolling, (like some Arch derivatives do) doesn't solve anything. I think I'm not wrong to say that roughly 90% of Arch Linux maintainance is not caused by mere breakage/bugs, but by continuous API/ABI/configuration changes during software upgrades. These problems would still hit "stable" - just a little later.

I am not saying that the 'stable' repository would be rolling. A released version is a released version, there would be no automatic rolling of new versions.

You could upgrade to the next version by bumping the version number inside your pacman.conf file.

Essentially, you would have three versions running at any time, similar to debian.

'old-stable', the last supported stable version
'stable', the current supported stable version
'unstable', the next stable version

When unstable moves to stable, it would be supported for say, 1.5 years. 12 months after stable is promoted from unstable, development would begin again on the new unstable.

Each of the versions would have their own testing repository, where patches can be tested by the community.

ayekat wrote:

You can't just start putting software packages into "freezed" state on a regular basis, keep backporting bug fixes for old software versions, test that the random selection of packages with a bunch of patches strapped onto them works flawlessly together, and recruit tons of people to build up an infrastructure for maintaining all that workflow (because that's the only correct way to do stable releases), while at the same time claim to be a "vanilla and KISS, yay!" distribution.

I agree that there would be a ton of work, especially w.r.t all the required infrastructure.

There is no doubt in my mind, however, that tools could definitely be built to simplify the process of maintaining version bumps. ie, watching repositories for new tags, helping to cherry pick bug & security patches, ensuring that new packages built by makepkg for stable repos do not do version bumps, and the only updates to the build scripts are patches that have been uploaded by developers.

fsckd · 2016-04-17 12:20:35

Why not use Debian then?

Edit: Maybe interested persons could create an "Archian" distribution. Essentially Debian repackaged to use pacman instead of dpkg. Debian has a ton of resources which makes it possible for them to maintain stable, testing, and unstable and do a fairly good job at it.

Last edited by fsckd (2016-04-17 12:26:57)

jryan · 2016-04-17 13:39:35

fsckd wrote:

Why not use Debian then?
Edit: Maybe interested persons could create an "Archian" distribution. Essentially Debian repackaged to use pacman instead of dpkg. Debian has a ton of resources which makes it possible for them to maintain stable, testing, and unstable and do a fairly good job at it.

That idea might actually be practical.

Arch-Hoochie · 2016-04-17 14:50:42

I've been using arch as a server for 3 years now been far more stable than my friends ubuntu setup. I am no pro though and it looks like I might need to rebuild mine now.

JohnBobSmith · 2016-04-17 17:46:07

What about decreasing rate of Pacman -Syu'ing? Instead of updating daily, update, say, monthly? Do it whenever you would regularly do system maintenance and have staff on hand. Then the administrator can weed out things like kernel upgrades very easily. IMO, and with no industry experience, that might work better than trying to mould an entire distro into something it wasn't meant to be.

Scimmia · 2016-04-17 17:50:54

JohnBobSmith wrote:

What about decreasing rate of Pacman -Syu'ing? Instead of updating daily, update, say, monthly?

Then you get no security updates until you do.

brebs · 2016-04-17 18:31:03

JohnBobSmith wrote:

What about decreasing rate of Pacman -Syu'ing? Instead of updating daily, update, say, monthly?

That's what happens *anyway*. It would be mad for a server sysadmin to update daily, because every update is a compromise between:

1. Risk of getting hacked (which requires both an exploit and a hacker)
2. Risk of breaking, in some surprising and unwelcome way, what previously worked fine. And then having to explain to your boss, or your boss' boss (who knows bugger-all about computers) why you've just wasted a morning (and caused unexpected & unwanted server downtime) fixing a problem which you yourself caused

triforce · 2016-04-20 13:56:34

Count me in if you need some help with the project. I like the idea.

Archian is also quite a clever idea, I would get involved with that as well.

jryan · 2016-04-20 14:14:02

Even though this is crazy idea, I think I'd be cool if someone figured out rolling release with bug fixes. Basically support partial upgrades. A user could sit on versions of packages as long as they want, but upgrade things if they need bug/security fixes.

If there any sane method in which that's possible with Arch? I imagine the only way to support partial upgrade would be to have ABS on your system and if you want to upgrade a single package, you'd have to make sure any packages that depend on that get recompiled as well?

What's the primary reasons we don't support partial upgrades now?

edit: added more context, formatting

Last edited by jryan (2016-04-20 14:15:30)

tho068 · 2016-04-20 15:07:54

noupgrade in pacman.conf ?

I think it will be problematic quickly due to dependencies though.

nbd · 2016-04-20 15:50:02

Important packages can be conserved in sandboxes along with their dependencies (a separate directory contains a package and all its dependencies) and run with systemd-nspawn. For some packages it's space consuming, though. E.g. version 41 of Firefox with all dependencies takes about a half GB. Although one sandbox can be used for conserving more than one package, in which case they will share some dependencies.

2ManyDogs · 2016-04-20 17:46:48

jryan wrote:

What's the primary reasons we don't support partial upgrades now?

This explains why partial upgrades are not supported: https://wiki.archlinux.org/index.php/Sy … nsupported

jryan · 2016-04-20 18:42:40

2ManyDogs wrote:

jryan wrote:
What's the primary reasons we don't support partial upgrades now?
This explains why partial upgrades are not supported: https://wiki.archlinux.org/index.php/Sy … nsupported

TLDR; you have would have to recompile all dependencies of that package when a package gets a bug/security fix.

2ManyDogs · 2016-04-20 19:23:57

jryan wrote:

TLDR; you have would have to recompile all dependencies of that package when a package gets a bug/security fix.

Really? One paragraph is too long for you to read? That's not really what it says.

Last edited by 2ManyDogs (2016-04-20 19:24:24)

Arch Linux

#1 2016-03-30 11:24:49

Arch Linux 'stable' repositories for servers?

#2 2016-03-30 12:32:36

Re: Arch Linux 'stable' repositories for servers?

#3 2016-03-30 12:40:19

Re: Arch Linux 'stable' repositories for servers?

#4 2016-04-01 16:36:53

Re: Arch Linux 'stable' repositories for servers?

#5 2016-04-01 18:21:10

Re: Arch Linux 'stable' repositories for servers?

#6 2016-04-01 19:05:16

Re: Arch Linux 'stable' repositories for servers?

#7 2016-04-01 19:35:45

Re: Arch Linux 'stable' repositories for servers?

#8 2016-04-01 20:34:00

Re: Arch Linux 'stable' repositories for servers?

#9 2016-04-12 20:35:46

Re: Arch Linux 'stable' repositories for servers?

#10 2016-04-12 20:57:09

Re: Arch Linux 'stable' repositories for servers?

#11 2016-04-12 21:13:55

Re: Arch Linux 'stable' repositories for servers?

#12 2016-04-17 07:38:55

Re: Arch Linux 'stable' repositories for servers?

#13 2016-04-17 12:20:35

Re: Arch Linux 'stable' repositories for servers?

#14 2016-04-17 13:39:35

Re: Arch Linux 'stable' repositories for servers?

#15 2016-04-17 14:50:42

Re: Arch Linux 'stable' repositories for servers?

#16 2016-04-17 17:46:07

Re: Arch Linux 'stable' repositories for servers?

#17 2016-04-17 17:50:54

Re: Arch Linux 'stable' repositories for servers?

#18 2016-04-17 18:31:03

Re: Arch Linux 'stable' repositories for servers?

#19 2016-04-20 13:56:34

Re: Arch Linux 'stable' repositories for servers?

#20 2016-04-20 14:14:02

Re: Arch Linux 'stable' repositories for servers?

#21 2016-04-20 15:07:54

Re: Arch Linux 'stable' repositories for servers?

#22 2016-04-20 15:50:02

Re: Arch Linux 'stable' repositories for servers?

#23 2016-04-20 17:46:48

Re: Arch Linux 'stable' repositories for servers?

#24 2016-04-20 18:42:40

Re: Arch Linux 'stable' repositories for servers?

#25 2016-04-20 19:23:57

Re: Arch Linux 'stable' repositories for servers?

Board footer