You are not logged in.

#1 2017-06-18 19:49:30

lapsio
Member
From: Warsaw
Registered: 2015-09-30
Posts: 50

Arch update strategy for production server

I'm using Arch linux with hardened kernal as KVM virtualization host. Arch doesn't support partial updates and rebooting virtualization host is not really that simple. So my question is - what's "the best" (as in one you would recommend) strategy for Arch virtualization host updates? System is partially isolated (no internet access apart from whitelisted update mirrors). I see following options:

- do not update until necessary due to new software installation / maintenance window. (~1 month)
- update frequently but do not reboot, run on old kernel.
- live kernel patching? I'm not sure if Arch supports that though.

Rebooting virtualization host requires hibernating all VMs and that takes a while and results in considerable downtime however VMs running here are not THAT mission critical so I guess I could afford lets say 30 min downtime at night once a month. But I wouldn't like to do it more frequently than that. Is it worth to update between reboots? Like once a week or something.

Last edited by lapsio (2017-06-18 19:51:50)

Offline

#2 2017-06-18 19:58:54

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: Arch update strategy for production server

Arch on production sounds like a bad idea, at least that's the average consensus here


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Offline

#3 2017-06-18 20:01:52

headkase
Member
Registered: 2011-12-06
Posts: 1,975

Re: Arch update strategy for production server

If you want Linux on a production machine then in my opinion you should be looking at Debian.  9, Stretch, was just released so it's only half out-of-date.  wink  Seriously though, Arch is meant to be maintained more than it sounds you are able.

Offline

#4 2017-06-18 20:03:47

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: Arch update strategy for production server

Only kernel updates and updates to external modules supplied by other packages should require a reboot.  All userspace software should not require a reboot.
I would not recommend updating the kernel package and then not rebooting as there is not benefit in such an update until you start the new kernel and the installed modules will mismatch with the old running kernel.
Not sure on live kernel patching and linux-hardened has kexec disabled.  If there was an exploitable kernel issue leaving it unpatched for several weeks may not be advisable either.
Would recommend regular updates for all software but kernel and kernel modules and track updates to the linux-hardened package and research what has changed in each update.
Edit:
grammar add missing not

Last edited by loqs (2017-06-18 20:04:29)

Offline

#5 2017-06-18 21:47:37

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,449
Website

Re: Arch update strategy for production server

Upating once a month is perfectly viable.  Many archers like updating much more frequently that that, but once-monthly should not lead to any problems.

And apparently my actions are directly against the above-mentioned alleged consensus.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#6 2017-06-18 22:40:44

lapsio
Member
From: Warsaw
Registered: 2015-09-30
Posts: 50

Re: Arch update strategy for production server

Okay, thank you for your thoughts on that topic smile

It's sad that grsecurity kernel is no longer supported as grsecurity closed, they seemed to have a bit more "lts" nature, but on the other hand I noticed significant stability improvement comparing to grsecurity kernel which sometimes misbehaved with KVM for some reason.

Last edited by lapsio (2017-06-18 22:45:03)

Offline

#7 2017-06-19 00:34:48

HiImTye
Member
From: Halifax, NS, Canada
Registered: 2012-05-09
Posts: 1,072

Re: Arch update strategy for production server

you can run the -lts kernel to make kernel updates less frequent, but you'll still want to reboot for, for instance driver and xorg updates (or at least re-load X for X updates). I've heard that you can do live kernel updates using kexec, but I've never personally tested it, as I don't mind having about a minute's worth of down time every few days.

Offline

#8 2017-06-19 01:05:23

lapsio
Member
From: Warsaw
Registered: 2015-09-30
Posts: 50

Re: Arch update strategy for production server

HiImTye wrote:

you can run the -lts kernel to make kernel updates less frequent, but you'll still want to reboot for, for instance driver and xorg updates (or at least re-load X for X updates). I've heard that you can do live kernel updates using kexec, but I've never personally tested it, as I don't mind having about a minute's worth of down time every few days.

Hmm... Xorg updates on headless virtualization server you say. I'll remember that during next update xD

linux-hardened iirc doesn't have -lts equivalent.

Last edited by lapsio (2017-06-19 01:05:52)

Offline

#9 2017-06-19 01:47:14

fsckd
Forum Fellow
Registered: 2009-06-15
Posts: 4,173

Re: Arch update strategy for production server

HiImTye wrote:

I've heard that you can do live kernel updates using kexec, but I've never personally tested it, as I don't mind having about a minute's worth of down time every few days.

Live updates is a different mechanism (see ksplice, kgraft, etc.). Kexec is more like fast rebooting. The existing userland does not stay after a kexec.


aur S & M :: forum rules :: Community Ethos
Resources for Women, POC, LGBT*, and allies

Offline

#10 2017-06-19 02:09:25

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Arch update strategy for production server

Trilby wrote:

Upating once a month is perfectly viable.  Many archers like updating much more frequently that that, but once-monthly should not lead to any problems.

And apparently my actions are directly against the above-mentioned alleged consensus.

I've got machines which are updated once a month or less. The key for me though is having other more regularly updated machines (and keeping an eye on the mailing list and forums) which serve as a rudimentary test system for updates.

That being said most show-stopping updates are recognizable way in advance in my experience, if you're on the mailing lists. You can then plan around them.

That being said.... if I was running an always-on system which couldn't tolerate downtime I'd just go with debian stable. Use the system most suitable for the task.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#11 2017-06-19 02:58:39

headkase
Member
Registered: 2011-12-06
Posts: 1,975

Re: Arch update strategy for production server

Also, for Debian, packages tend to get only security updates.  If you need bleeding-edge then you're usually make installing directly into your filesystem which is bad.  See: CheckInstall as kind of an equivalent to Debian for Arch's AUR.  With CheckInstall you can make Debian packages that you can manage with apt-get.  It's not perfect but depending on exactly what you need it could be damn close.

Offline

#12 2017-06-19 07:20:41

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Arch update strategy for production server

Too bad checkinstall fails in more than half of cases you throw at it... but it's more interesting to keep discussion about Arch than "use distro X because it's better for Y".

https://lists.archlinux.org/pipermail/a … 33791.html

Last edited by Alad (2017-06-19 07:20:54)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#13 2017-06-19 10:16:29

nbd
Member
Registered: 2014-08-04
Posts: 389

Re: Arch update strategy for production server

New program versions are untested and full of bugs. After any update you can get silently malfunctioning system.

Some time ago this bug has stopped all timers (cron jobs) on my machine, which was discovered only in some time:

https://bbs.archlinux.org/viewtopic.php?id=209572

There are many threads on this forum where using Arch in production is "officialy" discouraged.


bing different

Offline

#14 2017-06-19 10:52:44

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,449
Website

Re: Arch update strategy for production server

nbd wrote:

New program versions are untested and full of bugs.

This is possibly/occassionally true, but is absolutely absurd as a general conclusion.

This mentality reminds me of a freind of mine who worked on an "organic" farm.  It was her job, every morning, to go out and spray the crops with this "organic" pesticide.  After doing so for many months she saw how that sprayer was refilled: from a source container with all sorts of toxic warning labels on it.  She started asking questions and learned that this "organic" farm refused to use the newest generation of pesticides because they were not thoroughly tested and might contain dangers that we weren't yet aware of.  So instead they were using pesticides from last generation that no sane person would go anywhere near because we know exactly what dangers they contained: and they were horrible.

New software isn't perfect.  But to assume that the newer versions are as a rule more broken than the older versions they replace one would have to conclude that the developer(s) are just completely incompetent.  And if you are going to conclude that the developer(s) of a bit of software are completely incompetent, then really you shouldn't use any version of their software.

We do not have perfection anywhere in our world.  But we do have progress.  So your choice is not between perfect and imperfect but between either accepting or rejecting progress.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#15 2017-06-19 10:56:08

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Arch update strategy for production server

nbd wrote:

New program versions are untested and full of bugs. After any update you can get silently malfunctioning system.

Some time ago this bug has stopped all timers (cron jobs) on my machine, which was discovered only in some time:

https://bbs.archlinux.org/viewtopic.php?id=209572

There are many threads on this forum where using Arch in production is "officialy" discouraged.

It's entirely up to you if you deploy updates in production without testing them. The link above (which I assume you didn't read) says exactly that. The same issue can happen with a "stable" distribution, e.g. how Debian recently shipped a new, broken OpenSSL update to their stable branch even faster than Arch did.

Last edited by Alad (2017-06-19 10:56:44)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#16 2017-06-19 12:38:58

Mr.Elendig
#archlinux@freenode channel op
From: The intertubes
Registered: 2004-11-07
Posts: 4,092

Re: Arch update strategy for production server

Nothing special needed for running arch:

1. run your own repo
2. run test boxes
3. only push updates to your repo after you have tested them

This is what you should do no matter which distro you are running on your production machines.


Evil #archlinux@libera.chat channel op and general support dude.
. files on github, Screenshots, Random pics and the rest

Offline

#17 2017-06-19 13:31:15

adesh
Member
Registered: 2016-10-05
Posts: 167

Re: Arch update strategy for production server

Mr.Elendig wrote:

3. only push updates to your repo after you have tested them

This is what you should do no matter which distro you are running on your production machines.

Exactly, this is not specific to Arch. Any change you make on production server, be it code change, software upgrade, new hardware or change in network, should be not be done without proper testing. Treat production machines how they should be treated.

Offline

Board footer

Powered by FluxBB