Updating many (10+) Archlinux systems

a_neutrino · 2015-12-19 04:46:32

Hello,

I'd like to read about your recommendations and experience with maintaining a fleet of many (10+) Archlinux systems, physical or virtual. What is your experience with maintaining Archlinux in production? Do you think it scales?

I tried writing an Ansible playbook for system updates but either I'm doing it wrong or Archlinux updates really can't be semi-automated that way. With up to four systems I can split a tmux window and broadcast keystrokes in all panes at once which works well for pacman -Sc, pacman -Syu, rankmirrors and pacman-optimize. But I still have to go to each system and do pacdiff separately.

Optional context information

I'm looking to build a server at home which will virtualize an Archlinux system for each service using Xen and will hopefully be orchestrated with Ansible. There will be only 1 service per VM for maximum compartmentalization and security.

Here is my list of services I would like to run in separate VMs: Web, WebDAV, CalDAV, CardDAV, SMTP, Tor, OpenVPN, DHCP, VoIP, Backups, XMPP, LDAP, SQL, git, Nagios, Munin, DNS.

jasonwryan · 2015-12-19 05:57:36

Not a Sysadmin issue, moving to Network and Server...

nomorewindows · 2015-12-19 06:37:29

How about clusterssh?
You could run network pacman cache.
You could also run a proxy server.

Last edited by nomorewindows (2015-12-19 06:38:10)

rollenwiese · 2015-12-20 02:49:20

I have a metal VM Host running Arch. One of its libvirt guests runs Centos 7 serving Foreman/Puppet. I'm working on getting Foreman/Puppet set up to manage a collection of Arch machines offering different services.

I'm only the point where I have Foreman able to spin up a basic Arch machine with ntp/ssh and some custom user accounts.

Some yet to be written custom/modified puppet modules will have to handle the rest. Still learning how to do that though.

I'm pretty certain Foreman/Puppet can do what you want, but it's likely a long road to get there.

samlung · 2015-12-20 03:47:06

I was looking into managing Arch systems in a similar but not identical way but haven't gotten around to it. From what I could tell Ansible or SaltStack seemed the most appropriate for my needs. They already have extras for pacman:
http://docs.ansible.com/ansible/pacman_module.html
https://docs.saltstack.com/en/latest/re … acman.html
I guess for AUR packages you can use a yaourt wrapper, someone started one:
https://github.com/cdown/ansible-yaourt

In your case, orchestrating VMs from scratch sounds more like a job for Vagrant (or similar):
https://wiki.archlinux.org/index.php/Vagrant

There are so many of these tools nowadays, makes my head spin.

Last edited by samlung (2015-12-20 03:49:16)

a_neutrino · 2015-12-20 18:20:21

nomorewindows, thanks for the network pacman cache idea, good one. I'll add it to my wishlist.

rollenwiese and samlung, you mentioned good tools to automate the deployment and management of VMs and I'll keep those in mind.

For this thread, I was more interested to discuss the scalability of Archlinux in regards to OS updates with pacman. With CentOS and other versioned system, updates can be unmanned because they keep the APIs stable (yum update, etc.). With Arch, it is mandatory to read the output of pacman -Syu and take manual actions such as merge config files.

So how do you update, check the pacman -Syu log and merge .pacnew files on an increasingly large number of machines? Do you have tools, workflows, insights for that task?

Thank you for your time and attention.

samlung · 2015-12-20 19:51:47

in a well prepared pure VM setup, I think update breakage by pacman should be rare anomalies. I always keep an extra VM dedicated to testing updates which contains the union of all packages used on the other machines. the only differences not accounted for are network configs and some specific app profiles, but most differences are on the test machine in some form or another. if it works on the test machine, it's a go.

this is where I see the value in puppet/ansible/etc. they help ensure machines won't deviate from a state. I currently do too much manually and the risk of updates breaking mostly comes from that and unintentional/unaccounted for deviations in the single machines. I hope I can use ansible, and the test machine's ansible yaml will be the union of all the others.

none of this will work if you're in a very heterogenic hardware environment with software to go with it (like some labs?). in that case, clusterssh might actually become the best tool. but half the reason of using VMs is to avoid that.

for scalability (say over 15 machines) I can't give insight. probably calls for more thorough update testing setup, though then again, if it's all VMs with no AUR it's probably not that bad. you definitely want a local pacman repo (edit: or not quite, since it seems frowned upon/impractical in arch, but it's what I did for other OS; use package cache or this thing https://wiki.archlinux.org/index.php/Pacserve).

Last edited by samlung (2015-12-21 00:04:32)

samlung · 2015-12-20 20:04:03

(I said 15 machines but that's arbitrary, it's not so much about the number of machines but rather the heterogenity)

Arch Linux

#1 2015-12-19 04:46:32

Updating many (10+) Archlinux systems

Optional context information

#2 2015-12-19 05:57:36

Re: Updating many (10+) Archlinux systems

#3 2015-12-19 06:37:29

Re: Updating many (10+) Archlinux systems

#4 2015-12-20 02:49:20

Re: Updating many (10+) Archlinux systems

#5 2015-12-20 03:47:06

Re: Updating many (10+) Archlinux systems

#6 2015-12-20 18:20:21

Re: Updating many (10+) Archlinux systems

#7 2015-12-20 19:51:47

Re: Updating many (10+) Archlinux systems

#8 2015-12-20 20:04:03

Re: Updating many (10+) Archlinux systems

Board footer