Syncing multiple Archlinux machines

emadelsaid · 2024-01-30 00:07:55

I have 5 Archlinux machines I use frequently, they vary slightly in hardware (different thinkpad laptops, PC with different GPUs)
I need my systems to sync, installing a package in one system gets installed on the rest of the systems. setting configuration in one gets synced to the others...etc
The way I do it is ansible playbook, and dotfiles repository synced with Git. I change them, commit, push and when I get to use the other machines I pull and run the scripts that copies dotfiles/install packages...etc
The problem with this approach is that if I used `pacman` to install a package and didn't add it to the ansible playbook the system deviates. and the more commands I run that change the system the more it deviates from the rest.
What I would like to achieve is not just to install missing packages and copy over configuration but to ensure the whole system state is mostly identical (aside from hardware related stuff like fstab..etc)
I have noticed NixOS does this approach but it goes further to make a store of packages and override shell variables and in the process doesn't ahere to the linux filesystem anymore.
I like the idea of having a declarative way to describe the system. but I don't like the fact that nix doesn't have the normal linux filesystem. makes me confused. not my personal favorite
So I realized I would like something like NixOS but it operates on my current system. install/uninstall packages. goes over files and clean up anything that's not described in "the configuration/declaration". apply changes/modifications to files like setting keyboard layouts, timezone, locales, users, groups, services...etc

In the past days I tried to build a proof of concept but it's proving to be a bit hard. so I wanted to ask here what are the tools we have for Archlinux so far that can achieve this experience or at least can be glued together to achieve that.
Any tool that can provide part of that would help cut down the effort.

Thanks in advance.

Outswayer · 2024-02-02 04:35:32

I think you're pretty much on the right track with ansible and nix.

I'm dealing with exactly the same issue - i have 4 arch machines, 3 of them i use frequently. I also set up ansible to install them and sync config with git. But just like you, i sometimes install something manually just to test it out, and if i end up liking it i never actually put it into the ansible config. also, there's just so MANY config files, and adding them all to version tracking explicitly is a ton of work i haven't been willing to do, but syncing the entire home won't work because a lot of apps clutter all kinds of places with files you don't want to sync.

Nix seems like the far better option from what i can see - but it's a pretty steep learning curve, and a big initial effort to get it set up. i also feel like it has a lot of overhead in general, but i haven't actually worked with it much.

To keep systems in sync, my next idea would be to make ansible run regular checks of installed packages, and alert me if anything's not explicitly configured via ansible. and maybe a similar thing for dotfiles/home directory, just regularly run git status and exclude anything that pops up i don't want.

but honestly, it's a bunch of work, and i've kind of resorted to just having 1 main system for most of what i do, and leave the rest in "good enough" state... but it is holding me back sometimes. looking forward to any other input in this thread!

Allan · 2024-02-02 04:39:59

There have been tools written for this over the years... e.g.

https://lib.rs/crates/pacdef
https://github.com/rendaw/decpac

Outswayer · 2024-02-02 16:49:31

Thanks for those links, Allan!

from a quick glance, it seems they both just manage installed packages, which is a good start - but for my use case, i want to define the package and it's configuration in the same place. Luckily, decpac mentions another project:

https://github.com/CyberShadow/aconfmgr

that project seems like a nice way of doing things. it both manages the installed packages, and lets you specify which paths to keep track of, where it compares current config to whatever config is defined in it's files. i'll try setting it up on a fresh arch install and see how much effort it is to add and adjust things.

as a quick note, when running it on a system i've used for a while already, it tried to copy around 80'000 files to /tmp/. make sure to configure some more ignore paths, as the paths ignored by default don't seem to cut it.

Last edited by Outswayer (2024-02-02 16:51:36)

emadelsaid · 2024-02-08 08:59:55

Thanks for referring to these projects, Like any developer I didn't like 1 design decision that most of the projects take: "using configuration file instead of a programming language" maybe as a developer I expected to write in a programming language instead of YAML or JSON or in case of decpac Luxem.

So I spent some time in the past week writing a simple DSL for myself that can manage packages, install AUR packages, enable services/timers.

And this is what I have so far https://github.com/emad-elsaid/archlinu … uration.rb

Here are some design decisions I made:

Functions will accumulate what it needs to do and do them in bulk together. so writing `package 'curl'` then `package 'wget'` doesn't mean it'll install curl then install wget. but it'll install them in one step.
If I didn't use a function it's not going to run. so `package` will install the packages you pass to it and remove packages that you didn't pass to it. but if you didn't use this function it will not manage your packages. that applies to all other functions. and this is because the first time you call the function it'll register the step to install/uninstall packages and not registered globally. so the functions that are not used are not considered at all.
I went for a simple DSL similar to what the ruby community is used to.
because it's a ruby code you can include as many other methods you write as you wish. it's a ruby program that you run so that means you can use any ruby gem in your configuration or any function from the stdlib.

The project is very early and I broke my system once while developing it (removed all foreign packages by mistake). so please be careful if you decided to use it.

Arch Linux

#1 2024-01-30 00:07:55

Syncing multiple Archlinux machines

#2 2024-02-02 04:35:32

Re: Syncing multiple Archlinux machines

#3 2024-02-02 04:39:59

Re: Syncing multiple Archlinux machines

#4 2024-02-02 16:49:31

Re: Syncing multiple Archlinux machines

#5 2024-02-08 08:59:55

Re: Syncing multiple Archlinux machines

Board footer