You are not logged in.

#1 2018-10-13 20:12:49

zorggy
Member
Registered: 2013-05-30
Posts: 20

Scripting ArchLinux installation

Hello,

I'm really happy with ArchLinux and it's rolling release. I don't have to re-install my computers every 6 months as I did with Fedora during many years. And versions of programs are very bleeding-edge.

However, as I'm always discovering and testing programs and way of managing the system, the installations quickly become dirty and sometimes instable. Fortunately I have an Ansible playbook to set back most system parameters. But sometimes my changes are too nasty and I have to re-install. Arch-Linux does not provide any way of automation to quickly re-install. A solution would be to create a virtual host (KVM, for example) to realize my tests, but the image would need to be updated every day, as the host. I'm not fond of that solution. Another solution would be to script the installation. Of course, I know that maintainers may change the way to install at any time, but I see it's rather stable, and I'm ready to make my installation scripts evolving.

So I've created 2 Bash scripts, following the ArchLinux guide: One to format the system disk and install the basic packages, and one to configure, from chroot. I then get a very basic installation that I can specialize with my Ansible playbook. Great! Now, I would like to extract configuration data from the scripts so that I could tweak any type of installation just changing the config file, such as Anaconda with its kickstart file on Redhat distribution.

I decided to use the YAML format for configuration data. Here is an example:

---
config:
  pre_install_exec:
  post_install_exec: post-install.sh
  keyboard:
  bootmode: UEFI
  systemdisk:
    device: /dev/nvme0n1
    format: true
    table: gpt
    partitions:
    - device: /dev/nvme0n1p1
      mountpoint: /boot
      type: fat32
      size: 512
      label: boot
      boot: true
    - device: /dev/nvme0n1p2
      mountpoint: /
      type: ext4
      size: 32768
      label: linux
    - device: /dev/nvme0n1p3
      mountpoint: /opt
      type: ext4
      size: 32768
      label: opt
    - device: /dev/nvme0n1p4
      mountpoint: /home
      type: ext4
      size: 100%
      label: home
  mirrors:
  - https://archlinux.repo...
  - https://other_archlinux.repo...
  encrypted_root_password: letters_and_numbers...
  timezone: Continent/City
  localization:
  - en_US.UTF-8
  lang: en_US.UTF-8
  bootloader:
    type: systemd-boot
    timeout: 3
    title: ArchLinux
  fqdn: desktop.mydomain.com
  network:
    dns:
    - 192.168.1.1
    - 9.9.9.9
    domains:
    - mydomain.com
  finalreboot: true
...

But Bash has difficulties to interpret the YAML format (no dictionaries). So I tried with Python. I've not finished yet but it's almost working. However it calls many times just system commands (subprocess.Popen(...)). Moreover I need to install python and the yaml library in the new installation to run the second script (~145MB).

Do you have any idea on the subject? Using a C++ or Rust program? Or Bash with another format for the config data?

Thanks in advance for your answers.

Offline

#2 2018-10-13 20:28:04

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,306
Website

Re: Scripting ArchLinux installation

You should never have to reinstall arch.  Nuking the system and reinstalling from scratch (with or without a script) is robbing you of the opportunity to learn how to actually manage an arch system.

zorggy wrote:

the installations quickly become dirty and sometimes instable.

Define dirty.  Clean it.

Why would it be unstable?  Define unstable.

Last edited by Trilby (2018-10-13 20:28:20)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2018-10-13 20:30:00

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 26,070
Website

Re: Scripting ArchLinux installation

Moving to Programming and Scripting...


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#4 2018-10-14 16:20:46

zorggy
Member
Registered: 2013-05-30
Posts: 20

Re: Scripting ArchLinux installation

Hello,

As writing my first post, I realized I could give Bash a try, because I feel it's heavy to write the program in C++ and I don't know Rust even if I believe it's a good language.

So I've translated the installation config file from YAML to a Bash file that will have to be sourced. Here is the code:

#!/bin/bash
PRE_INSTALL_EXEC=
POST_INSTALL_EXEC=post-install.sh
KEYBOARD=
BOOTMODE=UEFI
SYSTEMDISK_DEVICE=/dev/nvme0n1
SYSTEMDISK_FORMAT=TRUE
SYSTEMDISK_TABLE=gpt
declare -a PARTITIONS # ( device mountpoint type sizeMiB label boot )
PARTITIONS[0]=(/dev/nvme0n1p1 /boot fat32 512 boot TRUE)
PARTITIONS[1]=(/dev/nvme0n1p2 / ext4 32768 linux FALSE)
PARTITIONS[2]=(/dev/nvme0n1p3 /opt ext4 32768 opt FALSE)
PARTITIONS[3]=(/dev/nvme0n1p4 /home ext4 100% home FALSE)
MIRRORS=('https://archlinux.repo...' 'https://other_archlinux.repo...')
ENCRYPTED_ROOT_PASSWORD='letters_and_numbers'
TIMEZONE=Continent/City
LOCALIZATION=(en_US.UTF-8)
DEFAULT_LANG=en_US.UTF-8
FQDN=desktop.domain.com
NETWORK_DNS=(192.168.1.1 9.9.9.9)
NETWORK_DOMAINS=(mydomain.com)
BOOTLOADER_TYPE=systemd-boot
BOOTLOADER_TIMEOUT=3
BOOTLOADER_TITLE=ArchLinux
FINAL_REBOOT=TRUE

It's not as readable as YAML format, in my opinion. But it's finally more compact. And no need to install python3 just to run the second script. I've already the Bash scripts but I now need to use the sourced variables instead of directly writing them in the code.

I'll write the different installation possibilities, such as BOOTMODE=bios, BOOTLOADER_TYPE=grub and provide the possibility to use NetworkManager instead of systemd-networkd.

I can finish this little project and be able to re-install in few minutes my hosts. Thank you!

Offline

#5 2018-11-09 06:18:06

thiagowfx
Member
Registered: 2013-07-09
Posts: 552

Re: Scripting ArchLinux installation

Trilby wrote:

You should never have to reinstall arch.  Nuking the system and reinstalling from scratch (with or without a script) is robbing you of the opportunity to learn how to actually manage an arch system.

zorggy wrote:

the installations quickly become dirty and sometimes instable.

Define dirty.  Clean it.

OP probably means: temporary files, cache, /var, hidden directories within /home (e.g. .local, .cache, .config), packages that are accumulated over time. Most operations are reversible but do require some thinking. For example, you can always nuke your $HOME and start over, but you can't do it on /var.

You can rely on a few things like paccache, bleachbit, systemctl reboot (ok, this one is just sarcasm). But it is not trivial to reset your system to a "clean" state without reinstalling it.

Offline

#6 2018-11-09 06:39:25

mpan
Member
Registered: 2012-08-01
Posts: 447
Website

Re: Scripting ArchLinux installation

zorggy: do you burn down your house, because you haven’t vacuumed it often enough? Don’t do the same to your system.

In particular don’t get caught in the endless “next time it will be perfect” loop. It will never be, because you repeat the same mistakes over and over again. Instead try to precisely define the dirtyness in your system. This way you break through the barrier you are probably dealing with now: seeing it as a huge, fuzzy, unattackable enemy. Then search how deal with it. This way you learn how to manage your system in a way that lets you keep it clean.

mpan, the owner of a 7+ years old Arch installation


Sometimes I’m a bit harsh — don’t get offended too easily!
PGP: 7C848198AE93D3BB | Coreutils SHA2 performance
Russian roulette: curl "https://ptpb.pw/$(cat /dev/urandom | tr -cd [:alnum:] | head -c 4)" | sudo bash

Offline

#7 2018-11-09 06:54:51

Awebb
Member
Registered: 2010-05-06
Posts: 5,202

Re: Scripting ArchLinux installation

I used to install Arch again in the beginning, because I kept doing noob mistakes and wanted to understand the process better. The second point isn't relevant anymore, because the AIF is gone and there is nothing intransparent about the process anymore. Today I only install Arch on new devices and virtual machines, which I frequently do, as I keep throwing away VM's when I'm done, because installing Arch is so easy and straight forward, that I don't bother keeping a base VM around.

Offline

#8 2018-11-09 13:59:45

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,306
Website

Re: Scripting ArchLinux installation

thiagowfx wrote:

Most operations are reversible but do require some thinking.

Yes.  Running arch linux requires thinking.  Anyone averse to this would be best served not by continually reinstalling arch, but rather by installing some other OS.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#9 2018-11-09 22:54:17

Roken
Member
From: UK
Registered: 2012-01-16
Posts: 904

Re: Scripting ArchLinux installation

Whilst experimenting with stuff, I've probably killed Arch completely half a dozen times since my first install (one time, I completely trashed permissions on rootfs - that was a doozy). To date, I haven't re-installed once. I've fixed what either went wrong, or what I did wrong. Why do we need a script?

Last edited by Roken (2018-11-09 22:56:12)


[img=Speedtest]http://www.speedtest.net/my-result/5145583518[/img]

Ryzen 1800x 8 core/16 thread - GTX 1060 6Gb, Asus ROG STRIX B350-F, 16Gb Corsair DDR4, Cooler Master N300 chassis, 6 HD (2SSD - 4Spinners) + 1 x optical.
Linux user #545703

Offline

#10 2018-11-11 13:05:45

simona1970
Banned
Registered: 2017-10-23
Posts: 10

Re: Scripting ArchLinux installation

I use a log .txt for system setup or modify. Exist a log of all packages installed, and utility to clean system like bleachbit. Forgotten something? And a I do gparted full backup of all partitions.

Last edited by simona1970 (2018-11-11 13:06:35)

Offline

#11 2018-11-11 20:18:37

thiagowfx
Member
Registered: 2013-07-09
Posts: 552

Re: Scripting ArchLinux installation

@mpan:That's a pretty good analogy. You beat me.

Trilby wrote:

Yes.  Running arch linux requires thinking.  Anyone averse to this would be best served not by continually reinstalling arch, but rather by installing some other OS.

Thinking is good. Toiling is not. I personally like the idea of a "clean.sh" script that is incremented over time that you run every few months or so. And then, every time you find something new to clean (according to your own use case and system), you add it to the script so you don't have to worry about it in the future. This prevents you from trying to "reinventing the wheel" every time a new cleaning operation is performed -- the definition of "cleaning" is open and subjective.

Offline

#12 2018-11-11 20:27:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,306
Website

Re: Scripting ArchLinux installation

thiagowfx wrote:

the definition of "cleaning" is open and subjective.

You litterally just defined it in your previous post as reinstalling the system which should never be necessary.

EDIT: *headdesk* thank for the correction bellow, and sorry for my poor initial reading.

Last edited by Trilby (2018-11-12 00:00:55)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#13 2018-11-11 21:57:10

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,175

Re: Scripting ArchLinux installation

Trilby wrote:
thiagowfx wrote:

the definition of "cleaning" is open and subjective.

You litterally just defined it in your previous post as reinstalling the system.  But hey, I guess I'm just trolling according to you, so don't mind me.

Pretty sure he wrote 'toiling' (I read it as trolling initially too).


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#14 2018-11-12 07:58:28

Awebb
Member
Registered: 2010-05-06
Posts: 5,202

Re: Scripting ArchLinux installation

I, too, read trolling. The internet isn't a good place to lighten a soul. Everyone becomes a cynic sooner or later.

Offline

#15 2018-11-14 05:42:21

thiagowfx
Member
Registered: 2013-07-09
Posts: 552

Re: Scripting ArchLinux installation

Lol, I meant toiling literally indeed, not trolling, it's not like I'm a random new user tongue

@Trilby: it's still subjective, the way I defined it is how I see it for me. I'll end this discussion here though because it's very open ended and I don't want to derail too much.

Another thing that I remembered these days was btrfs. You can be fancy by creating a bunch of partitions and immediately snapshotting them, and then "cleaning" your system later on by resetting some of its snapshots. I think I recall a few blog posts that preached that. Not very KISS though.

Offline

Board footer

Powered by FluxBB