Arch: taking it mainstream?

pgoetz · 2015-04-24 16:02:52

I run Arch on 3 production servers and my home server, which currently serves as the test machine for updates.

I think the server wiki page is not so much the issue. People who are setting up a server generally know what software they want to run, and if they don't, Arch is mostly certainly not the distro for them! The biggest issue for servers is stability. About a month ago there was a 2.x to 3.0 postfix update which required configuration file updates that weren't as clearly communicated as they could have been. I installed the updates on my home server; noticed that mail no longer worked and hit the forum to find out why. Then for the production servers I implemented the configuration file changes before doing the update: Production downtime = 0 seconds. Being able to patch kernels on the fly is also a big issue, as a lot of people can't readily reboot their servers. I was recently working on some Debian+CentOS servers at work that all hadn't been rebooted for 400-800 days (there's one server that's been up for 1500+ days!). Rebooting them required posting notices in several places a week in advance and scheduling a reboot time accurate to within 5 minutes. Given the frequency with which core software components are updated on Arch, it should be clear that this would become pretty burdensome for administrators under these conditions.

What's needed is a conceptual framework for how one can implement a rolling release model OS like Arch in a stable production environment. The reality is that we're being hit with security flaws that need to be patched at what seems to be an ever increasing rate, so just not updating servers and letting them run for years on end is no longer an option, either. So if you're going to have to update some things, why not just keep everything up to date? The big problem will be sofware components like apache, PHP, ruby on rails, and so on. These groups introduce updates which are not backwards compatible. Arch jettisons the old version for the new one as soon as the new one is proven to be stable in testing, and admins could be stuck with figuring out how to upgrade hundreds, and in some cases, thousands of servers which will break as soon as the update is applied. And even when the update is backwards compatible, it can still break locally created mission-critical software systems.

There is no obvious solution to these issues. What the major commercial distros currently do is probably optimal: fix the software for some period of time and backport security patches. This isn't really Arch's thing, but I would be hard pressed to think of a better base distribution to use for creating a stable distribution. Arch gives you the utmost flexibility in deciding when to introduce a package freeze for your stable distro, and doesn't come cluttered with a bunch of distro-specific bullshit, like Debian, for example (Redhat is even worse, don't know about SUSE).

Last edited by pgoetz (2015-04-24 16:05:07)

crouse · 2015-05-05 17:23:48

Your last two paragraphs are exactly why I don't use Arch Linux as a server any more. I'd prefer too, as I love the rolling release way of doing things, but in reality, it quickly becomes a nightmare in a production server environment. If you have time and resources to test all of your servers/services before applying updates, it's awesome, I did that for years. The point it becomes futile is when your servers are no longer all alike and you don't have a test environment that duplicates what you have in production and your upgrades proceed to break things. It can also get very problematic if you have hundreds or thousands of servers and all of the sudden your upgrade is going to require "manual intervention", at that point you better have your puppet and chef skills honed.

All distro's have their niche, I think Arch's is the "bleeding edge"... not sure I can even envision how that would make the leap to server distro.

lew21 · 2015-06-25 12:08:48

I'm running Arch dedicated servers for 5+ years, and was never able to keep up with its rolling release nature. I used to set it up with the latest packages, install everything I need, and not apply any updates. Every ~2 years I'd simply install a new Arch on a new server, migrate everything, and abandon the old one. But this way I don't get any security updates, and I have problems with adding new applications to the server. One time, I needed ffmpeg on a server that was 2 years old, so the easiest way to get it... was to install the latest version of Arch into a subdirectory, and run it in a chroot.

I don't think Arch is well suited to be the main server OS. Upgrades can break lots of things, especially booting, and an unbootable dedicated server is not a nice situation. I think it's a better idea to run something like CoreOS - because of its really safe update strategy ( https://coreos.com/using-coreos/updates/ ) and then run Arch instances in containers. This way, the main OS will never break, each container can be updated in isolation, and each container's upgrade can be tested before going live. We get all the benefits of rolling release systems, and no disadvantages. Arch really shines there.

pgoetz · 2015-06-25 15:27:42

Well, so I've come up with a solution to this problem, but as with most things computer-related it requires adding a level of indirection; in this case an intermediary between Arch and the production server. Let's call that intermediary Arch-Server. The idea is basically that Arch-Server using something like Mercurial or Git to maintain package trees for "point" stable releases at nearly any level of granularity, say weekly, which should be good enough. Let's say you install a new server on 2015-06-25. The Arch-Server project then keeps an image of the Arch package tree from that week which would only receive security patch updates, similar to other distributions. You can continue to use it without having to worry about some software update breaking your applications. It seems reasonable to maintain these point releases for 2-4 years. After that, the security patches stop being applied (upstream probably won't be bothering with versions this old) and it will be time to upgrade.

So then why not just use one of the standard commercial distros? Because you're still getting 100% up-to-date software on the day you install and you can incrementally move from one stable package tree to another whenever you like.

Last edited by pgoetz (2015-06-25 15:30:31)

progandy · 2015-06-25 17:48:42

pgoetz, so in extreme cases where a software gets a new release every week you want the maintainers to support between 104 and 208 versions of a single package...

dockland · 2015-06-25 19:46:38

I would love to have an Arch KDE LTS-kernel enviroment @ work.

pgoetz · 2015-06-25 20:31:45

progandy wrote:

pgoetz, so in extreme cases where a software gets a new release every week you want the maintainers to support between 104 and 208 versions of a single package...

I'm not sure what you mean by maintain in this context. The Arch-Server project would just take whatever is currently in Arch and preserve it under a time-stamped branch. The tricky part of this that I didn't get in to is applying security patches to potentially 100 versions of a package the git/mercurial tree. And for the record I know of no package which has a new release every month, let alone every week. However, this could still turn in to a chore. This is the part of the project that would require some automated wizardry, say using something like Jenkins to automate security patch compiles.

I think it would have to be understood that if upstream is no longer supporting a particular version of a package then the Arch-Server project couldn't, either. That's completely within the bounds of the Arch philosophy. In practice, most major packages will continue to supply patches for at least a couple of years of releases.

progandy · 2015-06-25 21:19:59

pgoetz wrote:

I'm not sure what you mean by maintain in this context. The Arch-Server project would just take whatever is currently in Arch and preserve it under a time-stamped branch. The tricky part of this that I didn't get in to is applying security patches to potentially 100 versions of a package the git/mercurial tree. And for the record I know of no package which has a new release every month, let alone every week. However, this could still turn in to a chore. This is the part of the project that would require some automated wizardry, say using something like Jenkins to automate security patch compiles.

Well, I exaggerated the systemd cycle a bit. As far as I know they don't support old versions anymore, only the most recent few.

If you create a snapshot every week, then with 500 packages you could get at least one package with a new version in each snapshot. Let's assume you get a critical patch for glibc. Now you'll have to check everything, since this might (in a very unlikely case, but still) change the behaviour of glibc in a very subtle way and somewhere might be a package combination in a package tree that doesn't like this change, and not everything can be tested in an automated environment. Even if it's possible you might need a few hundred hours to test everything, maybe only half an hour per package tree, but with 200+ package trees that accumulates.

pgoetz · 2015-06-25 21:30:40

Yep, this would definitely be a cutting edge project with some real complications/hurdles, but I think we're at the point where this is within reach. Arch-Server would likely need to be monetized with something more than t-shirt sales on Zazzle; say by offering support contracts like Canonical/Red Hat do in order to have a few full time people working on this.

Also, a bug in glibc is sort of a worst case scenario, I agree.

Last edited by pgoetz (2015-06-25 21:37:17)

jasonwryan · 2015-06-25 21:33:48

pgoetz wrote:

say by offering support contracts like Canonical/Red Hat do in order to have a few full time people working on this.

We have a dedicated support service already: allan@archlinux.org

pgoetz · 2015-06-25 21:39:51

jasonwryan wrote:

We have a dedicated support service already: allan@archlinux.org

I'm willing to bet that Allan is not going to be interested in maintaining 200 simultaneous Arch branches; at least not by himself. <:)

Spider.007 · 2015-06-26 08:47:54

pgoetz wrote:

[...]
About a month ago there was a 2.x to 3.0 postfix update which required configuration file updates that weren't as clearly communicated as they could have been. I installed the updates on my home server; noticed that mail no longer worked and hit the forum to find out why.

I think this is the key issue. I have also found that major upgraded don't get the attention / precaution they deserve. I understand archlinux developer cannot be expected to know about all the possible breakages that major upgrades introduce; but there have been multiple occasions where packages that cannot work at all have been released; as well as updates that required manual configuration updates that were not communicated. This is probably caused by developers that don't actually use the packages they maintain.

It would be great if others in the community could help improve the quality of these upgrades. It could be as simple as participating in the testing repos, and reporting bugs.

pgoetz wrote:

Being able to patch kernels on the fly is also a big issue, as a lot of people can't readily reboot their servers. I was recently working on some Debian+CentOS servers at work that all hadn't been rebooted for 400-800 days (there's one server that's been up for 1500+ days!). Rebooting them required posting notices in several places a week in advance and scheduling a reboot time accurate to within 5 minutes. Given the frequency with which core software components are updated on Arch, it should be clear that this would become pretty burdensome for administrators under these conditions.

I have found that the longer you don't restart a server, the harder it gets to actually do it. I have also seen machines with uptime in the 500+ days, and my fix was:

* redundancy; you don't want a single machine doing anything that other machines cannot do. Not only a single machine but also whole rack or an entire datacenter should be designed for redundancy
* automated fallover; whenever something stops working; fallover to another machine

once you have this in place; it's easy to reboot machines without anyone noticing.

pgoetz wrote:

What's needed is a conceptual framework for how one can implement a rolling release model OS like Arch in a stable production environment. The reality is that we're being hit with security flaws that need to be patched at what seems to be an ever increasing rate, so just not updating servers and letting them run for years on end is no longer an option, either. So if you're going to have to update some things, why not just keep everything up to date? The big problem will be sofware components like apache, PHP, ruby on rails, and so on. These groups introduce updates which are not backwards compatible. Arch jettisons the old version for the new one as soon as the new one is proven to be stable in testing, and admins could be stuck with figuring out how to upgrade hundreds, and in some cases, thousands of servers which will break as soon as the update is applied. And even when the update is backwards compatible, it can still break locally created mission-critical software systems.

I do not recognize problems in any of the components you mention; all of those also know their userbase and have compatibility in mind when releasing new versions. If you're concerned about that you should upgrade your development/testing environment more often, find issues early and participate in the arch/upstream bugtrackers

pgoetz · 2015-06-26 16:30:27

Spider.007 wrote:

I do not recognize problems in any of the components you mention; all of those also know their userbase and have compatibility in mind when releasing new versions. If you're concerned about that you should upgrade your development/testing environment more often, find issues early and participate in the arch/upstream bugtrackers

Well, I can tell you from personal experience that there have been PHP updates which have broken custom applications on servers that I run -- I have had to spend many hours vi'ing through PHP files updating code that someone else wrote. However, I'm a bit unclear on what point you're trying to make. Are you suggesting that there's not really a need for a stable package base for servers?

Arch Linux

#26 2015-04-24 16:02:52

Re: Arch: taking it mainstream?

#27 2015-05-05 17:23:48

Re: Arch: taking it mainstream?

#28 2015-06-25 12:08:48

Re: Arch: taking it mainstream?

#29 2015-06-25 15:27:42

Re: Arch: taking it mainstream?

#30 2015-06-25 17:48:42

Re: Arch: taking it mainstream?

#31 2015-06-25 19:46:38

Re: Arch: taking it mainstream?

#32 2015-06-25 20:31:45

Re: Arch: taking it mainstream?

#33 2015-06-25 21:19:59

Re: Arch: taking it mainstream?

#34 2015-06-25 21:30:40

Re: Arch: taking it mainstream?

#35 2015-06-25 21:33:48

Re: Arch: taking it mainstream?

#36 2015-06-25 21:39:51

Re: Arch: taking it mainstream?

#37 2015-06-26 08:47:54

Re: Arch: taking it mainstream?

#38 2015-06-26 16:30:27

Re: Arch: taking it mainstream?

Board footer