Scripting ArchLinux installation

zorggy · 2018-10-13 20:12:49

Hello,

I'm really happy with ArchLinux and it's rolling release. I don't have to re-install my computers every 6 months as I did with Fedora during many years. And versions of programs are very bleeding-edge.

However, as I'm always discovering and testing programs and way of managing the system, the installations quickly become dirty and sometimes instable. Fortunately I have an Ansible playbook to set back most system parameters. But sometimes my changes are too nasty and I have to re-install. Arch-Linux does not provide any way of automation to quickly re-install. A solution would be to create a virtual host (KVM, for example) to realize my tests, but the image would need to be updated every day, as the host. I'm not fond of that solution. Another solution would be to script the installation. Of course, I know that maintainers may change the way to install at any time, but I see it's rather stable, and I'm ready to make my installation scripts evolving.

So I've created 2 Bash scripts, following the ArchLinux guide: One to format the system disk and install the basic packages, and one to configure, from chroot. I then get a very basic installation that I can specialize with my Ansible playbook. Great! Now, I would like to extract configuration data from the scripts so that I could tweak any type of installation just changing the config file, such as Anaconda with its kickstart file on Redhat distribution.

I decided to use the YAML format for configuration data. Here is an example:

---
config:
  pre_install_exec:
  post_install_exec: post-install.sh
  keyboard:
  bootmode: UEFI
  systemdisk:
    device: /dev/nvme0n1
    format: true
    table: gpt
    partitions:
    - device: /dev/nvme0n1p1
      mountpoint: /boot
      type: fat32
      size: 512
      label: boot
      boot: true
    - device: /dev/nvme0n1p2
      mountpoint: /
      type: ext4
      size: 32768
      label: linux
    - device: /dev/nvme0n1p3
      mountpoint: /opt
      type: ext4
      size: 32768
      label: opt
    - device: /dev/nvme0n1p4
      mountpoint: /home
      type: ext4
      size: 100%
      label: home
  mirrors:
  - https://archlinux.repo...
  - https://other_archlinux.repo...
  encrypted_root_password: letters_and_numbers...
  timezone: Continent/City
  localization:
  - en_US.UTF-8
  lang: en_US.UTF-8
  bootloader:
    type: systemd-boot
    timeout: 3
    title: ArchLinux
  fqdn: desktop.mydomain.com
  network:
    dns:
    - 192.168.1.1
    - 9.9.9.9
    domains:
    - mydomain.com
  finalreboot: true
...

But Bash has difficulties to interpret the YAML format (no dictionaries). So I tried with Python. I've not finished yet but it's almost working. However it calls many times just system commands (subprocess.Popen(...)). Moreover I need to install python and the yaml library in the new installation to run the second script (~145MB).

Do you have any idea on the subject? Using a C++ or Rust program? Or Bash with another format for the config data?

Thanks in advance for your answers.

Trilby · 2018-10-13 20:28:04

You should never have to reinstall arch. Nuking the system and reinstalling from scratch (with or without a script) is robbing you of the opportunity to learn how to actually manage an arch system.

zorggy wrote:

the installations quickly become dirty and sometimes instable.

Define dirty. Clean it.

Why would it be unstable? Define unstable.

Last edited by Trilby (2018-10-13 20:28:20)

jasonwryan · 2018-10-13 20:30:00

Moving to Programming and Scripting...

zorggy · 2018-10-14 16:20:46

Hello,

As writing my first post, I realized I could give Bash a try, because I feel it's heavy to write the program in C++ and I don't know Rust even if I believe it's a good language.

So I've translated the installation config file from YAML to a Bash file that will have to be sourced. Here is the code:

#!/bin/bash
PRE_INSTALL_EXEC=
POST_INSTALL_EXEC=post-install.sh
KEYBOARD=
BOOTMODE=UEFI
SYSTEMDISK_DEVICE=/dev/nvme0n1
SYSTEMDISK_FORMAT=TRUE
SYSTEMDISK_TABLE=gpt
declare -a PARTITIONS # ( device mountpoint type sizeMiB label boot )
PARTITIONS[0]=(/dev/nvme0n1p1 /boot fat32 512 boot TRUE)
PARTITIONS[1]=(/dev/nvme0n1p2 / ext4 32768 linux FALSE)
PARTITIONS[2]=(/dev/nvme0n1p3 /opt ext4 32768 opt FALSE)
PARTITIONS[3]=(/dev/nvme0n1p4 /home ext4 100% home FALSE)
MIRRORS=('https://archlinux.repo...' 'https://other_archlinux.repo...')
ENCRYPTED_ROOT_PASSWORD='letters_and_numbers'
TIMEZONE=Continent/City
LOCALIZATION=(en_US.UTF-8)
DEFAULT_LANG=en_US.UTF-8
FQDN=desktop.domain.com
NETWORK_DNS=(192.168.1.1 9.9.9.9)
NETWORK_DOMAINS=(mydomain.com)
BOOTLOADER_TYPE=systemd-boot
BOOTLOADER_TIMEOUT=3
BOOTLOADER_TITLE=ArchLinux
FINAL_REBOOT=TRUE

It's not as readable as YAML format, in my opinion. But it's finally more compact. And no need to install python3 just to run the second script. I've already the Bash scripts but I now need to use the sourced variables instead of directly writing them in the code.

I'll write the different installation possibilities, such as BOOTMODE=bios, BOOTLOADER_TYPE=grub and provide the possibility to use NetworkManager instead of systemd-networkd.

I can finish this little project and be able to re-install in few minutes my hosts. Thank you!

thiagowfx · 2018-11-09 06:18:06

Trilby wrote:

You should never have to reinstall arch. Nuking the system and reinstalling from scratch (with or without a script) is robbing you of the opportunity to learn how to actually manage an arch system.
zorggy wrote:
the installations quickly become dirty and sometimes instable.
Define dirty. Clean it.

OP probably means: temporary files, cache, /var, hidden directories within /home (e.g. .local, .cache, .config), packages that are accumulated over time. Most operations are reversible but do require some thinking. For example, you can always nuke your $HOME and start over, but you can't do it on /var.

You can rely on a few things like paccache, bleachbit, systemctl reboot (ok, this one is just sarcasm). But it is not trivial to reset your system to a "clean" state without reinstalling it.

mpan · 2018-11-09 06:39:25

zorggy: do you burn down your house, because you haven’t vacuumed it often enough? Don’t do the same to your system.

In particular don’t get caught in the endless “next time it will be perfect” loop. It will never be, because you repeat the same mistakes over and over again. Instead try to precisely define the dirtyness in your system. This way you break through the barrier you are probably dealing with now: seeing it as a huge, fuzzy, unattackable enemy. Then search how deal with it. This way you learn how to manage your system in a way that lets you keep it clean.

mpan, the owner of a 7+ years old Arch installation

Awebb · 2018-11-09 06:54:51

I used to install Arch again in the beginning, because I kept doing noob mistakes and wanted to understand the process better. The second point isn't relevant anymore, because the AIF is gone and there is nothing intransparent about the process anymore. Today I only install Arch on new devices and virtual machines, which I frequently do, as I keep throwing away VM's when I'm done, because installing Arch is so easy and straight forward, that I don't bother keeping a base VM around.

Trilby · 2018-11-09 13:59:45

thiagowfx wrote:

Most operations are reversible but do require some thinking.

Yes. Running arch linux requires thinking. Anyone averse to this would be best served not by continually reinstalling arch, but rather by installing some other OS.

Roken · 2018-11-09 22:54:17

Whilst experimenting with stuff, I've probably killed Arch completely half a dozen times since my first install (one time, I completely trashed permissions on rootfs - that was a doozy). To date, I haven't re-installed once. I've fixed what either went wrong, or what I did wrong. Why do we need a script?

Last edited by Roken (2018-11-09 22:56:12)

simona1970 · 2018-11-11 13:05:45

I use a log .txt for system setup or modify. Exist a log of all packages installed, and utility to clean system like bleachbit. Forgotten something? And a I do gparted full backup of all partitions.

Last edited by simona1970 (2018-11-11 13:06:35)

thiagowfx · 2018-11-11 20:18:37

@mpan:That's a pretty good analogy. You beat me.

Trilby wrote:

Yes. Running arch linux requires thinking. Anyone averse to this would be best served not by continually reinstalling arch, but rather by installing some other OS.

Thinking is good. Toiling is not. I personally like the idea of a "clean.sh" script that is incremented over time that you run every few months or so. And then, every time you find something new to clean (according to your own use case and system), you add it to the script so you don't have to worry about it in the future. This prevents you from trying to "reinventing the wheel" every time a new cleaning operation is performed -- the definition of "cleaning" is open and subjective.

Trilby · 2018-11-11 20:27:07

thiagowfx wrote:

the definition of "cleaning" is open and subjective.

You litterally just defined it in your previous post as reinstalling the system which should never be necessary.

EDIT: *headdesk* thank for the correction bellow, and sorry for my poor initial reading.

Last edited by Trilby (2018-11-12 00:00:55)

ngoonee · 2018-11-11 21:57:10

Trilby wrote:

thiagowfx wrote:
the definition of "cleaning" is open and subjective.
You litterally just defined it in your previous post as reinstalling the system. But hey, I guess I'm just trolling according to you, so don't mind me.

Pretty sure he wrote 'toiling' (I read it as trolling initially too).

Awebb · 2018-11-12 07:58:28

I, too, read trolling. The internet isn't a good place to lighten a soul. Everyone becomes a cynic sooner or later.

thiagowfx · 2018-11-14 05:42:21

Lol, I meant toiling literally indeed, not trolling, it's not like I'm a random new user

@Trilby: it's still subjective, the way I defined it is how I see it for me. I'll end this discussion here though because it's very open ended and I don't want to derail too much.

Another thing that I remembered these days was btrfs. You can be fancy by creating a bunch of partitions and immediately snapshotting them, and then "cleaning" your system later on by resetting some of its snapshots. I think I recall a few blog posts that preached that. Not very KISS though.

qurn · 2019-03-26 19:09:19

I made a similar arch script. https://github.com/qurn/myarch/

My reasons were:

By writing a script you still learn about your system. In fact when you learn something, you can adjust your script and document it for every future install.
You can isolate bugs better with a reproducible system.
I switch computers on a regular basis, e.g. at work. And i don't want to spend hours of installation, optimization and might forget something like ucode and never notice it.
Suppose your system has a weird bug, maybe with xorg. Maybe you don't have the nerve to fix this and not the time to ask the forum.
I went through some dependency-hells and installed some unnecessary packages explicitly and might carry them around. Having the option to reinstall the system allows some bold approaches.
Programs leave files with weird names. I wish i didn't have to learn what .purple or trolltec means.
Isn't it cool to walk around the world and have your fully personalized arch just 20 min. away from every pc with internet.

Arch Linux

#1 2018-10-13 20:12:49

Scripting ArchLinux installation

#2 2018-10-13 20:28:04

Re: Scripting ArchLinux installation

#3 2018-10-13 20:30:00

Re: Scripting ArchLinux installation

#4 2018-10-14 16:20:46

Re: Scripting ArchLinux installation

#5 2018-11-09 06:18:06

Re: Scripting ArchLinux installation

#6 2018-11-09 06:39:25

Re: Scripting ArchLinux installation

#7 2018-11-09 06:54:51

Re: Scripting ArchLinux installation

#8 2018-11-09 13:59:45

Re: Scripting ArchLinux installation

#9 2018-11-09 22:54:17

Re: Scripting ArchLinux installation

#10 2018-11-11 13:05:45

Re: Scripting ArchLinux installation

#11 2018-11-11 20:18:37

Re: Scripting ArchLinux installation

#12 2018-11-11 20:27:07

Re: Scripting ArchLinux installation

#13 2018-11-11 21:57:10

Re: Scripting ArchLinux installation

#14 2018-11-12 07:58:28

Re: Scripting ArchLinux installation

#15 2018-11-14 05:42:21

Re: Scripting ArchLinux installation

#16 2019-03-26 19:09:19

Re: Scripting ArchLinux installation

Board footer