Arch: taking it mainstream?

pgoetz · 2015-03-07 11:05:32

I''ve been a linux user/admin for a very long time and have been using/experimenting with Arch for about a year. I love everything about the Arch philosophy and consider pacman to be vastly superior to either rpm or dpkg. I've even installed Arch on a couple of production servers, and this has been going well as well. Here is the problem. Whenever I mention that I'm using Arch in production I get comments like this

If you are running Arch in production, rather than something like CentOS or Debian Stable, you should probably be fired for gross incompetence.

(http://www.reddit.com/r/linux/comments/ … or/cp6ymbb)

The reason for this is quite sound, actually:

No sysadmin worth his salt would risk the liability of running important services on a "bleeding edge" distribution that has been known to break when just updating itself over the years. That's why corporate environments go with stable distributions that don't change unless they need to change, and that maintain binary compatibility for many years. Any admin in a large organization is going to be familiar with ancient hardware running ancient software because there is some mission critical device/program which hasn't been (or can't) be updated.
If Ruby 2.1.0 changes their implementation so it is not binary compatible with Ruby 2.0.0, then upgrading to 2.1.0 may break C extensions compiled against 2.0.0. Red Hat knows this, so they closely monitor every update for binary compatibility (and would likely stay at Ruby 2.0.0 -- even if they have to support it with custom security patches!). Arch doesn't monitor this, and doesn't care if it breaks, which is one reason why it's crap for anything that requires long-term stability.

In practice I've found Arch to be pretty damn stable, but then I've only been using it for a year. The really issue is the one raised concerning ABI/API changes. In many production environments organizations can't afford to deal with API changes in real time; i.e. so don't want to upgrade volatile stuff like PHP, Ruby, or even Apache.

What would be really cool is a system like Ubuntu, but based on Arch. This way you can keep test servers and desktops completely up to date while retaining stability for production systems with complex software environments.

Any thoughts on this? Maybe the solution is to figure out a better way to firewall older systems in order to address security exploits which are constantly being uncovered?

Alad · 2015-03-07 11:11:49

I suppose you could use a snapshot via Seblu's ARM, follow CVEs and keep it updated yourself through ABS. Whether the convenience of pacman makes up for the added maintenance effort I don't know.

Last edited by Alad (2015-03-07 11:12:55)

karol · 2015-03-07 12:30:28

Feel free to expand https://wiki.archlinux.org/index.php/Se … _server_OS

Last edited by karol (2015-03-07 12:32:00)

pgoetz · 2015-03-07 12:38:53

That would be an awful lot of work. And it's not just the convenience of pacman. As I mentioned, I've been using linux systems for a long time. Baked into the Debian philosophy is that some 14-year-old package maintainer has a better idea of where files should be installed than the upstream developer. This has annoyed me for years and became intolerable once I started using Arch; there's just no reason for this nonsense. Red Hat similarly moves crap around and invents all kinds of "microtechnologies" which are frequently not well thought out and cause more headache than help.

Short of an entirely new stable distro based on Arch, here is my proposal: how about flagging packages that break ABI/API compatibility so that admins can avoid installing them by rote; i.e.

# pacman -Syu

would work as it does now.

# pacman -Syu --no-api-changes

would only install packages that aren't guaranteed to break existing code; e.g. the upgrade from apache 2.2 to 2.4, or updates to PHP/Ruby which are not backwards compatible.

pgoetz · 2015-03-07 12:40:32

karol wrote:

Feel free to expand https://wiki.archlinux.org/index.php/Se … _server_OS

Thanks for alerting me to the existence of that page! Every time I think I'm reasonably familiar with the Wiki (and believe me, many of my hours have gone into editing it), I find out that there's stuff I had no idea existed.

Edit: yep, that page needs a lot of work. So many Wiki updates, so little time....

Last edited by pgoetz (2015-03-07 12:42:14)

pgoetz · 2015-03-07 12:57:10

There are no obvious solutions to my query. If there were a perfect system out there, everyone would be using it, and Microsoft has few qualms about breaking .NET API compatibility, for example, which is why most linux + windows admins I know approach upgrading Windows servers with great trepidation.

Right now, I think an api-change flag added to pacman would swing the pendulum in favor of Arch, all things considered. Here is my follow up comment on reddit, which provides justification for this claim:

I've been managing hundreds of linux systems for almost 20 years, and the Arch package management and philosophy are vastly superior to debian/red hat. In practice we have the most problems with Debian stable systems, because users actually need new software features, so admins take it upon themselves to "hybridize" the system with packages from Debian testing/experimental. This then becomes a true maintenance nightmare. Red Hat similarly suffers from outdated software, although I'll admit that I haven't used Red Hat enterprise software. I have seen a less experience sys admin from our umbrella IT group reduced to tears because she was unable to install Red Hat on a cutting edge new server because the necessary drivers just weren't there.
Ubuntu had the right idea; I just wish someone would do something like Ubuntu, but based on Arch instead of Debian, which I've come to view as complete and utter crap put together by amateurs. Let me give you one example. I believe you will agree that the functionality of hardware RAID is somewhat important? With the exception of LSI, all raid hardware appears to be complete crap, and we have on and off problems with LSI as well. After one such sequence of problems I dug into the current kernel source and realized that the LSI drivers were at least a couple of revs behind what LSI had available. Meanwhile the current Ubuntu distro was several rev's behind that, and Ubuntu LTS was several revs behind that. So if I wanted my hardware to be stable, I'm stuck installing custom kernels. Yes, yes, Ubuntu makes newer kernels available via PPA, but Ubuntu is based on Debian, so this is trying to make a silk purse out of a sow's ear.

Allan · 2015-03-07 13:14:45

The was an Arch Server project for a while. It failed due to no-one really being that interested.

smirky · 2015-03-07 13:20:32

I would also like to see Arch making mature steps which might provide some guidelines for maintaining a production server. This is actually not an easy thing, since it requires people to look into the new software and give a lot of time and effort into new packages. Also even if that kind of testing does happen, it's not enough guarantee for your system not to break, since there might be something in the latest version which is totally wrong but then again, transparent to testing. In order this to happen, the community should get bigger and much more devoted to the cause, which isn't easy at all. That's why some of those devs that maintain server versions of a distribution are on a payroll (at least that's how I heard).

Last edited by smirky (2015-03-07 13:21:23)

pgoetz · 2015-03-07 13:21:31

Allan wrote:

The was an Arch Server project for a while. It failed due to no-one really being that interested.

I think a lot of people (including myself) were sucked in by the Ubuntu hype. Ubuntu has steadily gotten worse and more unstable (witness the switchover to systemd from upstart that is going to occur in a few days, never mind Wayland vs. Mir on the desktop), while Arch has gotten steadily better and more stable over time.

Bottom line is the time might be right to reboot the Arch server project. Does anyone have any references to this effort?

pgoetz · 2015-03-07 13:25:17

smirky wrote:

Also even if that kind of testing does happen, it's not enough guarantee for your system not to break, since there might be something in the latest version which is totally wrong but then again, transparent to testing.

In reality, this is true about every distribution, not to mention MS Windows. The admins I know who manages linux and Windows servers are generally more terrified about upgrading Windows, as apparently Microsoft frequently breaks .NET API compatibility. At our weekly admin meetings I here about Windows server horror stories all the time; really bizarre stuff like having to write VB scripts to delete all the cached AD user accounts every 30 minutes or the users can't log in again after some amount of time has passed.

There are no perfect solutions, and it seems quite unlikely that there will be any. Because people.

Last edited by pgoetz (2015-03-07 13:27:33)

tomk · 2015-03-07 13:53:44

IMO if you do stuff to Arch to make it a server/"production" distro, then it's not Arch anymore.

Anyway, archserver.org no longer exists, but the wayback machine returns this.

ANOKNUSA · 2015-03-07 14:06:41

pgoetz wrote:

Short of an entirely new stable distro based on Arch, here is my proposal: how about flagging packages that break ABI/API compatibility so that admins can avoid installing them by rote...

Sysadmins don't want to be glorified bug testers, and the Arch devs don't need the burden of supporting businesses with all that extra free labor. That's exactly why Arch is not suitable for critical enterprise datacenters.

EDIT: Also, the difference between what you're proposing and a new distro isn't very great.

Last edited by ANOKNUSA (2015-03-07 14:07:33)

pgoetz · 2015-03-07 22:46:10

ANOKNUSA wrote:

EDIT: Also, the difference between what you're proposing and a new distro isn't very great.

I agree with all of your comments. Unfortunately there is no perfect solution to this problem.

karol · 2015-03-07 22:58:15

If you want more comments, read https://bbs.archlinux.org/viewtopic.php?id=162434 and http://www.reddit.com/r/archlinux/comme … n_servers/

karol · 2015-03-08 17:52:15

See also the epic https://bbs.archlinux.org/viewtopic.php … 44#p612344

pgoetz · 2015-03-09 05:55:55

tomk wrote:

Anyway, archserver.org no longer exists, but the wayback machine returns this.

It appears that some of the people behind the archserver project are still around. archive.org appears to have only archived the front page. I wonder if anyone has the stuff from the wiki around. It would be interesting to see what kinds of ideas they were kicking around.

chaonaut · 2015-03-09 12:45:17

good news about kernel
No reboot patching comes to Linux 4.0
(but what about systemd updates?)

Last edited by chaonaut (2015-03-09 12:47:57)

progandy · 2015-03-09 13:45:15

chaonaut wrote:

(but what about systemd updates?)

systemd itself can be easily reloaded with the daemon-reexec command. Services must implement some kind of seamless restart if you want it, otherwise you have short interruptions. An example would be nginx. I'm not sure how well that works together with PIDFile= in systemd, though. http://wiki.nginx.org/CommandLine#Upgra … On_The_Fly

Edit: That doesn't make an arch server more stable, though. You still have the same software to test and verify, only reboots may be less frequent if arch supplies runtime upgrade packages.

Last edited by progandy (2015-03-09 13:54:26)

pgoetz · 2015-03-09 17:19:46

progandy wrote:

That doesn't make an arch server more stable, though. You still have the same software to test and verify, only reboots may be less frequent if arch supplies runtime upgrade packages.

True. I think ANOKNUSA is correct; the only way to accomplish this is with something that basically amounts to a new distro. I'm seriously thinking about rebooting the archserver project in order to provide canned server solutions to non-profits. I contacted one of the former archserver organizers to see if anyone still had the old archserver documentation around. It would be interesting to see what kinds of ideas they were kicking around.

I currently run Arch on 2 production servers, but one of them is my home server, which doesn't have anything on it that can't be down for a couple of hours. I stage updates to my own server first, make sure everything is OK, and then upgrade the "real" server running Arch. At work we have a large enough operation that there is a testing machine group. If we were to roll out Arch (and I'm the only Arch enthusiast in the group currently), pacman -Syu would be run daily only on the testing machines, with updates to the production machines coming from a local Arch repository that is on a 2 week delay from source. This way updates can be tested for a couple of weeks on the test machines before being rolled out to production.

Of course the problem with this is that you're almost certainly not going to test every use case. PHP or ruby gets updated and it seems fine on your test install, only to create problems on the production servers which have real users on them. The only way I can think to deal with this is to set up a very quick and easy to use rollback system. Of course this all gets considerably more complicated when you're using a configuration management system like puppet or ansible, because the rollbacks will need to take these into account as well. I haven't looked at how other people are handling rollback, but I could fairly easily write a front end to pacman that would make note of the version numbers of each package to be upgraded, making sure these exact packages are locally available and then another script to downgrade if there are problems. I'm assuming this is roughly how the other rollback systems work.

Allan · 2015-03-10 00:30:33

pgoetz wrote:

I currently run Arch on 2 production servers, but one of them is my home server, which doesn't have anything on it that can't be down for a couple of hours. I stage updates to my own server first, make sure everything is OK, and then upgrade the "real" server running Arch. At work we have a large enough operation that there is a testing machine group. If we were to roll out Arch (and I'm the only Arch enthusiast in the group currently), pacman -Syu would be run daily only on the testing machines, with updates to the production machines coming from a local Arch repository that is on a 2 week delay from source. This way updates can be tested for a couple of weeks on the test machines before being rolled out to production.

This does nothing for security issues, whose fixes may get rolled out as a package version update.

pgoetz · 2015-03-10 09:41:25

Allan wrote:

This does nothing for security issues, whose fixes may get rolled out as a package version update.

No, of course not. You're then stuck either trying to install just the package which has the security patch or hand patching the source and installing the binaries over the installed package. This is what long term release distros do, but they also roll it up into a security patch package. I will say that Arch's transparency should make the latter (hand patching) considerably easier (I haven't had to do it yet on Arch).

A rollback should be considered a temporary solution until the source of the problem can be identified and either fixed or addressed. A perfect example is a couple of weeks ago the roundcube maintainer rolled out an update which broke a lot of people's roundcube installs, including mine. Using the bug tracker as a workspace, a few of us worked together and had the problem resolved in 3-4 hours (the package needed some additional dependencies due to changes in how roundcube is coded). Unfortunately some of my users were without mail service for 4 hours. A trigger-configured rollback system would have solved this.

This does not address the problem of a complex software install that the organization can't or can't afford to update. On the other hand, there is no solution for this situation long term on any platform. Even the commercial distros only guarantee 4 years of security patches, and even then it's only select releases, which get long in the tooth rather quickly. Microsoft provides security patches for a considerably longer time, but also does point releases that often break peoples' code.

We do not live in a perfect world.

Last edited by pgoetz (2015-03-10 09:45:12)

speedyx · 2015-03-10 10:47:07

In my little world I recommend to use pkgclip or other cache utility to keep only one or two more old packages on the system. That permitted me to have always the way to downgrade without waste space. A full roll back system was never a necessity.

pgoetz · 2015-03-10 12:27:30

speedyx wrote:

In my little world I recommend to use pkgclip or other cache utility to keep only one or two more old packages on the system. That permitted me to have always the way to downgrade without waste space. A full roll back system was never a necessity.

What do you do if packages A, B, and C are upgraded. A needs to be rolled back, but B depends on A? What if A depends on B, and old A is incompatible with B?

exidux · 2015-04-18 21:35:10

chaonaut wrote:

good news about kernel
No reboot patching comes to Linux 4.0
(but what about systemd updates?)

A patching system for a running kernel could open it up a lot faster than before to loading bogus modules or patch data, but increases the ease for systems needing a lot of uptime. I guess this stresses the need for good firewall settings and passwords. I wonder if servers truly need such a thing.

Last edited by exidux (2015-04-18 21:36:15)

thoffmeyer · 2015-04-24 08:43:41

karol wrote:

Feel free to expand https://wiki.archlinux.org/index.php/Se … _server_OS

I love Arch Linux, and I run it on my server, albeit not a full production server, but I still run it on my server nonetheless. Maybe I will give this a go and try to.. work on the wiki page.. I will try to get to it this week

Arch Linux

#1 2015-03-07 11:05:32

Arch: taking it mainstream?

#2 2015-03-07 11:11:49

Re: Arch: taking it mainstream?

#3 2015-03-07 12:30:28

Re: Arch: taking it mainstream?

#4 2015-03-07 12:38:53

Re: Arch: taking it mainstream?

#5 2015-03-07 12:40:32

Re: Arch: taking it mainstream?

#6 2015-03-07 12:57:10

Re: Arch: taking it mainstream?

#7 2015-03-07 13:14:45

Re: Arch: taking it mainstream?

#8 2015-03-07 13:20:32

Re: Arch: taking it mainstream?

#9 2015-03-07 13:21:31

Re: Arch: taking it mainstream?

#10 2015-03-07 13:25:17

Re: Arch: taking it mainstream?

#11 2015-03-07 13:53:44

Re: Arch: taking it mainstream?

#12 2015-03-07 14:06:41

Re: Arch: taking it mainstream?

#13 2015-03-07 22:46:10

Re: Arch: taking it mainstream?

#14 2015-03-07 22:58:15

Re: Arch: taking it mainstream?

#15 2015-03-08 17:52:15

Re: Arch: taking it mainstream?

#16 2015-03-09 05:55:55

Re: Arch: taking it mainstream?

#17 2015-03-09 12:45:17

Re: Arch: taking it mainstream?

#18 2015-03-09 13:45:15

Re: Arch: taking it mainstream?

#19 2015-03-09 17:19:46

Re: Arch: taking it mainstream?

#20 2015-03-10 00:30:33

Re: Arch: taking it mainstream?

#21 2015-03-10 09:41:25

Re: Arch: taking it mainstream?

#22 2015-03-10 10:47:07

Re: Arch: taking it mainstream?

#23 2015-03-10 12:27:30

Re: Arch: taking it mainstream?

#24 2015-04-18 21:35:10

Re: Arch: taking it mainstream?

#25 2015-04-24 08:43:41

Re: Arch: taking it mainstream?

Board footer