Arch Setup Script Thoughts

GosuSan · 2016-02-17 07:23:00

Hey everyone,

I recently did 2 new setups with ArchLinux (a small Homeserver for a Friend and Dual-Boot on my Laptop).
This was the day where I thought about creating a post-Installation setup-script.

I prefer to do the installation itself manually because I don't want to give away control there and get used to this process.
But when I reach the point where the base system is ready and I have to set up everything, I'd like to automate this via a bash script.

Some things I plan to automate:

- creating new user ( + clone some basic config files from my github)
- installation of additional packages
- editing the config files (I think via sed? But what happens when some configuration files change over time? Maybe some sort of error-management)

- (optional step) install and configure DE

Some things I plan to NOT automate:

- new users password

What is your opinion on this? advantages and disadvantages? anything I did not think of?

EDIT:
It's mostly the part of config-files editing that worries me a bit. I know how to edit them via sed, but I don't know how to handle it when config files change in the future.
Is this even common? Or do config files not change much over time?

Thanks in advance!

Last edited by GosuSan (2016-02-17 07:35:05)

olive · 2016-02-17 07:46:20

The problem is that you cannot fully automate these tasks. You need the name of the new use, this is is a user choice that cannot be automated. Installation of additional packages cannot be automated because you do not know what the user want. Same for configuring the DE, this is obviously not possible to automate.

If you want to automate things for yourself, the simplest and most efficace method would be simply to create a tar file of the whole filesystem of one installation and then untar this on your new machine. You will need to reinstall the boot loader and be careful with the UUID of the filesystem. If you really want several partitions, you will have to make several tar files (but IMHO one big partition is easier and more convenient).

x33a · 2016-02-17 07:46:21

GosuSan wrote:

EDIT:
It's mostly the part of config-files editing that worries me a bit. I know how to edit them via sed, but I don't know how to handle it when config files change in the future.
Is this even common? Or do config files not change much over time?

There is no way to tell if and when the config files change or not. Arch ships upstream configurations and if the configs change upstream, they will be reflected in Arch Linux. Basically it means that if you intend to automate Arch installation by means of a script, you have to ensure that the script keeps working. There can never be a fire and forget solution due to the rolling nature of Arch.

GosuSan · 2016-02-17 08:07:21

olive wrote:

If you want to automate things for yourself, the simplest and most efficace method would be simply to create a tar file of the whole filesystem of one installation and then untar this on your new machine. You will need to reinstall the boot loader and be careful with the UUID of the filesystem. If you really want several partitions, you will have to make several tar files (but IMHO one big partition is easier and more convenient).

I do want to automate the setup after the installation, not the whole installation. And it is only for myself, If I screw something up.
EDIT: Also, a tarball of a whole system is much much bigger than a small script that I just can clone from my git and run it.
Also I may not want a DE on every installation, but some. So I think a script is more versatile.

x33a wrote:

There is no way to tell if and when the config files change or not. Arch ships upstream configurations and if the configs change upstream, they will be reflected in Arch Linux. Basically it means that if you intend to automate Arch installation by means of a script, you have to ensure that the script keeps working. There can never be a fire and forget solution due to the rolling nature of Arch.

Jep that is the main "problem". But maybe no "fire-and-forget" solution but an attempt to change the desired lines in my config files and feedback if this fails - with an message "config file xyz needs manual editing" or something like that?
Since these scripts are meant only for me and the use on my systems, this should be an solution?

Last edited by GosuSan (2016-02-17 08:10:13)

dsana123 · 2016-02-17 20:00:11

I have bash scripts to automate my Arch install in conjunction with git where I keep my configs (system and user). The install is only for myself so I can hard-code things like username. One bash script to install within the arch-chroot and to create a minimal environment, then a bash script to run after the arch-chroot that installs my graphical environment and packages.

I have a pretty normal system config so I can keep config files separate (e.g. I use /etc/system/journald.conf.d/custom.conf) without needing to modify the system configs.

I have a script to backup my config to my git repo so I always have the latest state (including the list of pacman packages), and my install scripts pull the data from my online git repo as part of the install process.

x33a · 2016-02-18 05:01:07

GosuSan wrote:

Jep that is the main "problem". But maybe no "fire-and-forget" solution but an attempt to change the desired lines in my config files and feedback if this fails - with an message "config file xyz needs manual editing" or something like that?
Since these scripts are meant only for me and the use on my systems, this should be an solution?

I feel that you should just start writing the script and modify it when and if any problem occurs. You are probably overthinking it

GosuSan · 2016-02-18 05:59:40

x33a wrote:

I feel that you should just start writing the script and modify it when and if any problem occurs. You are probably overthinking it

I guess you are right I really tend to overthink stuff a lot.
I have a script going now, I guess I'll test it a bit in an VM.

Thanks for your advices

brontosaurusrex · 2016-02-18 07:32:14

dsana123 wrote:

I have a pretty normal system config so I can keep config files separate (e.g. I use /etc/system/journald.conf.d/custom.conf) without needing to modify the system configs.

Could you explain the "keeping config files separate" part?

highend · 2016-02-18 08:34:01

dsana123 wrote:

I have bash scripts to automate my Arch install in conjunction with git where I keep my configs (system and user). The install is only for myself so I can hard-code things like username. One bash script to install within the arch-chroot and to create a minimal environment, then a bash script to run after the arch-chroot that installs my graphical environment and packages.
I have a pretty normal system config so I can keep config files separate (e.g. I use /etc/system/journald.conf.d/custom.conf) without needing to modify the system configs.
I have a script to backup my config to my git repo so I always have the latest state (including the list of pacman packages), and my install scripts pull the data from my online git repo as part of the install process.

Could you share them (as a starting point for new users like me)? Ofc after removing any personal information out of them...

dsana123 · 2016-02-18 19:13:57

brontosaurusrex wrote:

dsana123 wrote:
I have a pretty normal system config so I can keep config files separate (e.g. I use /etc/system/journald.conf.d/custom.conf) without needing to modify the system configs.
Could you explain the "keeping config files separate" part?

Sorry, I should have been more explicit. I meant I keep my own system custom config files separate from the system-generated config files. So, for example, if I wanted to customise the journald configuration I would not modify the /etc/systemd/journald.conf file, but instead create a /etc/system/journald.conf.d/custom.conf file where the configuration overrides would live. 'man journald.conf', or the Arch Linux Wiki, gives more information.

The seperation works well for system configs, but not at all well for daemon configs such as /etc/openvpn/server.conf. In this case I always backup these configs to my git repo when the configs change.

dsana123 · 2016-02-18 19:35:32

highend wrote:

dsana123 wrote:
I have bash scripts to automate my Arch install in conjunction with git where I keep my configs (system and user). The install is only for myself so I can hard-code things like username. One bash script to install within the arch-chroot and to create a minimal environment, then a bash script to run after the arch-chroot that installs my graphical environment and packages.
I have a pretty normal system config so I can keep config files separate (e.g. I use /etc/system/journald.conf.d/custom.conf) without needing to modify the system configs.
I have a script to backup my config to my git repo so I always have the latest state (including the list of pacman packages), and my install scripts pull the data from my online git repo as part of the install process.
Could you share them (as a starting point for new users like me)? Ofc after removing any personal information out of them...

Well, OK, but the script really just automates the steps you would go through manually (so why do I have a script? Because 1) I didn't know whether to install to SSD or SD card, and 2) I wanted to check Arch Linux stability on the laptop on a USB stick before committing, and 3) doing repetitive things annoy me).

Here's the script I invoke from inside arch-chroot to install the base system for my Macbook Pro (let the code-mocking commence).

https://gist.github.com/dsana123/bdc9a5d8ceb0f683fd36

The second bash script to install my graphical environment is simply more of the same (basically transcribing my notes into a script), with a few extras like installing AUR packages and personal packages from an up-to-date list of packages stored in my git repo. Perhaps I can leave that as a good learning exercise?

Last edited by dsana123 (2016-02-18 19:39:50)

highend · 2016-02-18 22:11:55

Thank you dsana123 for sharing the script!

Perhaps I can leave that as a good learning exercise?

Sure

One last question: How do you get your files from git during the setup process?
pacman -S git (don't know if the arch cd has it already)
and then checkout your arch install files repository?

dsana123 · 2016-02-18 22:23:24

highend wrote:

Thank you dsana123 for sharing the script!
Perhaps I can leave that as a good learning exercise?
Sure
One last question: How do you get your files from git during the setup process?
pacman -S git (don't know if the arch cd has it already)
and then checkout your arch install files repository?

See lines 51-64 of my script.
Line 52 does the pacman installation of git by installing over the network (if git is not already installed in the chroot).
Line 59 clones your git repo that contains the config files (this 'checks out' your git repo).

highend · 2016-02-19 09:54:11

See lines 51-64 of my script.

Uh, my bad, sorry!

Thanks again for sharing it, a great start to customize my future arch installations

Arch Linux

#1 2016-02-17 07:23:00

Arch Setup Script Thoughts

#2 2016-02-17 07:46:20

Re: Arch Setup Script Thoughts

#3 2016-02-17 07:46:21

Re: Arch Setup Script Thoughts

#4 2016-02-17 08:07:21

Re: Arch Setup Script Thoughts

#5 2016-02-17 20:00:11

Re: Arch Setup Script Thoughts

#6 2016-02-18 05:01:07

Re: Arch Setup Script Thoughts

#7 2016-02-18 05:59:40

Re: Arch Setup Script Thoughts

#8 2016-02-18 07:32:14

Re: Arch Setup Script Thoughts

#9 2016-02-18 08:34:01

Re: Arch Setup Script Thoughts

#10 2016-02-18 19:13:57

Re: Arch Setup Script Thoughts

#11 2016-02-18 19:35:32

Re: Arch Setup Script Thoughts

#12 2016-02-18 22:11:55

Re: Arch Setup Script Thoughts

#13 2016-02-18 22:23:24

Re: Arch Setup Script Thoughts

#14 2016-02-19 09:54:11

Re: Arch Setup Script Thoughts

Board footer