ArchBlocks - minimal/modular install framework [PURE SYSTEMD SUPPORT]

altercation · 2012-09-25 19:58:03

ArchBlocks - a minimal, modular, manual install framework

UDPATES:
16 Oct 2012 - Further systemd support, more queries in sample filesystem block for swap size, new sensors block support
09 Oct 2012 - Pure systemd support via new block. INIT_MODE variable (see sample installer) determines if used.
08 Oct 2012 - Added interactive install drive query if INSTALL_DRIVE is blank in config, Allow for non /dev/sda EFI setup
07 Oct 2012 - Less fragile at password entry points in script (allows user to try again on entry error); "mr" config support; tested on new 201210 install media
04 Oct 2012 - Was blowing up the /tmp tmpfs with AUR installs and heredocs. Made it bigger.
29 Sep 2012 - SIGNIFICANT structural changes (now uses subdirs) and cleanup - SEE BELOW (install file is very minimal now)
26 Sep 2012 - allow both EFI on EFI booted arch install media and now EFI install on *non* EFI booted arch install media / PXE

ArchBlocks is a very lightweight framework for creating quick, modular, *non-interactive* Arch Linux install scripts tailored to specific systems. It is accessible via a single quick curl downloadable script (from, for example, github).

WHERE:

Example repo at https://github.com/altercation/archblocks

EXAMPLE:

Boot Arch install media (in the example config I expect an EFI boot, but it's trivial to make this work for non EFI):

# curl -sfL https://raw.github.com/altercation/archblocks/master/example.sh" > install.sh; bash install.sh

or in short url form:

# curl -sfL http://git.io/eXaMPL3 > install.sh; bash install.sh

(this then sources the blocks of code remotely and configures the system based on the specific blocks called and variables set in it; see below)

WHAT IT ISN'T:

ArchBlocks is not an interactive general purpose script that asks you what you'd like. You *must* customize it (or its blocks) as much as any other install script. ArchBlocks simply makes this process easier to maintain as a single project that is applicable for multiple systems.

NOTE: It is possible to create interactive blocks, though I'm leaving this for later and focusing on getting some good default blocks in place first.

WHY:

Many somewhat experienced Arch users end up building lightweight install scripts to set up their systems. I've been wanting to modularize my own install scripts such that multiple, different system types (laptop/server/desktop/headless) can reuse most of the common elements easily and swap out blocks of code as needed.

AND YET it's easy to make something like this far too complex. Arch is Simple so Our Utilities Are Simple. ArchBlocks kickstarts off a *single* "configuration" script and loads blocks from a *single* directory (remote or local, doesn't matter). I've previously written extensive additions to AIF before it was frozen in carbonite, and that experience convinced me of the value of non-interactive, simple install script based approach.

NO REALLY, WHY NOT JUST MONOLITHIC INSTALL SCRIPTS, WHY MODULAR:

There is install code that may be impacted by changes to Arch, or I sometimes want to improve the way I handle some part of the installation. This may impact multiple systems. Rather than update and maintain five different manual install scripts, modular is a better approach. A simple example is the various `NETWORK_*` blocks. I can keep using the default network setup method in most of my scripts and use a light weight / minimal network install method by just swapping out a block. This minimal install script can be revised and all systems that I install with it in future will make use of the revision.

DETAILS:

Each system I install gets a single config file (the "tau" script in the example above is a laptop). This script loads a few very lightweight helper functions (maintained in a lib file for reuse across "config" scripts) and then loads blocks from the (surprise) blocks directory (local or remote). These blocks are almost all just simple blocks of bash script. The only exceptions should be very clear from the example: there are a few functions that are very specific to the filesytem configuration that must be sprinkled throughout the code. These functions and their intended sequence in the script are easily identified by name such as FILESYSTEM_PRE_BASEINSTALL (in this case FILESYSTEM_PRE_BASEINSTALL is either a null function or does something as defined in the FILESYSTEM block).

A config/install script (the only script you need to manually execute) looks like this (and nothing more... this is the actual script I use to install Arch on a Thinkpad x220):

REMOTE=https://raw.github.com/altercation/archblocks/master #e.g. file://.
HOSTNAME=tau
USERNAME=es
USERSHELL=/bin/bash
FONT=Lat2-Terminus16
LANGUAGE=en_US.UTF-8
KEYMAP=us
TIMEZONE=US/Pacific
MODULES="dm_mod dm_crypt aes_x86_64 ext2 ext4 vfat intel_agp drm i915"
HOOKS="base udev autodetect pata scsi sata usb usbinput consolefont encrypt filesystems fsck"
KERNEL_PARAMS="quiet" # set/used in FILESYSTEM,INIT,BOOTLOADER blocks (this gets added to)

HARDWARE=hardware/laptop/lenovo/thinkpad_x220
TIME=common/time_ntp_utc
FILESYSTEM=filesystem/gpt_luks_passphrase_ext4
BOOTLOADER=bootloader/efi_gummiboot
NETWORK=network/wired_wireless_minimal
AUDIO=common/audio_alsa
VIDEO=common/video_intel
POWER=common/power_acpi
#XORG=xorg/default.sh
#DESKTOP=desktop/xmonad_minimal
POSTFLIGHT="common/sudo_default common/create_user"
APPSETS="appsets/cli_utils appsets/edu_utils appsets/vim_core appsets/mutt_core"

. <(curl -fsL "${REMOTE}/blocks/lib/helpers.sh"); _loadblock lib/core

The blocks subdirectory contains blocks of simple bash script and looks something like this (note the variants in items like NETWORK... I am working on alternates for FILESYSTEM and BOOTLOADER as well, but the principle should be clear)::

└── blocks
    ├── _lib
    ├── appsets
    ├── bootloader
    │   ├── bios_grub2.sh
    │   └── efi_gummiboot.sh
    ├── common
    ├── desktop
    ├── filesystem
    │   ├── gpt_luks_passphrase_ext4.sh
    │   └── mbr_default.sh
    ├── hardware
    │   ├── desktop
    │   ├── laptop
    │   ├── peripheral
    │   └── server
    ├── network
    │   ├── wired_wireless_default.sh
    │   └── wired_wireless_minimal.sh
    └── xorg

That's it, really. Nothing fancy. Comprehensible, reusable, modular.

If you want to use something like this, fork away. Not for everyone, but hopefully useful for those that like this approach.

Last edited by altercation (2012-10-16 21:20:07)

Awebb · 2012-09-25 20:47:27

Eh, I really like this one. It looks like this could work. I will first read all the code and then try it to set up my mediacenter.

altercation · 2012-09-25 20:51:19

I'm really happy to take feedback and help you walk through getting it to work as well, including drafting some more sample blocks. A couple more sample blocks will make this more broadly functional. Message me here or directly (@ethanschoonover / es@ethanschoonover.com). -es

altercation · 2012-09-26 22:00:04

I've just updated the script to use the new gummiboot package from extra and to fallback to the default EFI bootloader method if being installed on a system that wasn't properly booted into EFI mode.

drobole · 2012-09-27 12:56:57

Looks like a nice backend.

Have you thought about making the disk/partition setup more configurable?

1. Installing alongside an existing OS
2. Use only % of disk
3. Separate home, dev, sys, ... partitions (Yes/No questions)
4. Configurable partition size for root and maybe others
5. Optional swap partition.

And how about configuration of multiple network interfaces (wired, wireless, mobile broadband)?

Another thing that might be cool is if I could keep a separate configuration file that I could feed into the installer. That way I could initiate the installation process on several machines really fast based on my pre-made configuration file

Last edited by drobole (2012-09-27 13:04:01)

altercation · 2012-09-27 23:49:00

drobole wrote:

Looks like a nice backend.
Have you thought about making the disk/partition setup more configurable?
1. Installing alongside an existing OS
2. Use only % of disk
3. Separate home, dev, sys, ... partitions (Yes/No questions)
4. Configurable partition size for root and maybe others
5. Optional swap partition.

I had initially intended the configuration to be purely done by selecting different blocks and hadn't intended this to turn into an interactive installer *but* it is possible that it could be interactive. One could even mix non-interactive with interactive blocks, or allow some basic interactivity in the main install script (install_tau.sh in my example repo). I want to be careful about how/if that would be added in to this framework since I don't want it become AIF level complex

As long as complexity is limited, I could make an example block with some queries in it (e.g. an initial query of the variables that are defined in the installer). Maybe I'll mock this up in a branch for consideration.

drobole wrote:

And how about configuration of multiple network interfaces (wired, wireless, mobile broadband)?

There are two NETWORK_* blocks in the current repo and I am working on a NETWORK_wired_static.sh as an example block for a static IP server style installation. I'd love to get more examples of these blocks contributed to the project.

drobole wrote:

Another thing that might be cool is if I could keep a separate configuration file that I could feed into the installer. That way I could initiate the installation process on several machines really fast based on my pre-made configuration file

Right now the goal is that the main install script (install_tau.sh) is also the config (in that the "stack" of blocks determines the configuration). Another option would be to create default install script that one only needs to override values in, but that is a slightly different approach. I'll consider how that might relate to the current form of this framework.

(note that I did previously create a project called backpac - https://github.com/altercation/backpac - that was designed to create config files for use during install/reinstall of Arch, so there is some overlap with that thinking)

Last edited by altercation (2012-09-27 23:50:14)

altercation · 2012-09-30 07:54:24

Just a note for those subscribed to this thread: I've updated the first post with details on a significant revision to the archblocks script that should make it much easier to configure. https://bbs.archlinux.org/viewtopic.php … 7#p1166527

The short of it is that I've implemented subdirectories and cleaned up the main install script (the part that will differ most significantly between systems). Additionally, this paves the way for interactive blocks if there is interest.

altercation · 2012-10-04 17:24:53

Updated the script to remount a larger tmpfs on /tmp to handle larger temp contents (installing packages from AUR, heredocs, etc.).

enduser · 2012-10-06 17:45:35

Hey, this looks pretty cool. I'm confused about a couple of things though: in the example script it says "boot into Arch Install media and run:

curl https://raw.github.com/altercation/archblocks/master/install_tau.sh" > install.sh; bash install.sh

However install_tau.sh doesn't seem to exist. Also, if the point of

curl url > out.sh; bash out.sh

is to ensure the script is fully downloaded before it's run (as opposed to `curl url | bash`, which (iiuc) runs the script as it's downloaded), shouldn't you use

curl url > out.sh && bash out.sh

to ensure the `curl` command didn't fail before running the script? Just a small thing, but I thought I'd ask.
Awesome project by the way, just what I'm looking for (unattended installation). I look forward to testing/making my own blocks soon (for instance an 'install my dotfiles' one)!

altercation · 2012-10-07 05:16:35

enduser wrote:

Hey, this looks pretty cool. I'm confused about a couple of things though: in the example script it says "boot into Arch Install media and run:
curl https://raw.github.com/altercation/archblocks/master/install_tau.sh" > install.sh; bash install.sh
However install_tau.sh doesn't seem to exist.

I've renamed the sample installer a couple times (right now it's example.sh). I'll update that information. I had named it install_tau.sh originally since I'm testing this on my ThinkPad which is named tau. I may rename this to "sample_laptop.sh" or some such later on...

It's worth noting that I structured this so that, if there was a big enough "library" of blocks, a user could keep just the config file (this example.sh installer) in their own repo and could source everything else from, for example, my master repo.

enduser wrote:

Also, if the point of
curl url > out.sh; bash out.sh
is to ensure the script is fully downloaded before it's run (as opposed to `curl url | bash`, which (iiuc) runs the script as it's downloaded), shouldn't you use
curl url > out.sh && bash out.sh
to ensure the `curl` command didn't fail before running the script? Just a small thing, but I thought I'd ask.

Yes, that's totally correct. I actually didn't want people doing that (so probably should just break that sample code out into two lines) since I still want them to peek at the installer first, particularly since there is a lot that happens automatically (and while there is a warning of DOOM upfront, it's always nice to make people type a bit more to think things through ;-) ).

enduser wrote:

Awesome project by the way, just what I'm looking for (unattended installation). I look forward to testing/making my own blocks soon (for instance an 'install my dotfiles' one)!

Excellent! Please do consider submitting any blocks you feel are general enough for group consumption back to the project as pull requests.

I'm actually adding in an "install my dotfiles" block now, but it's based on my using mr and a simply symlinking script in my dotfiles repo. It's a slick and easy way to do it, if anyone is interested.

altercation · 2012-10-08 05:32:29

FYI thread subs - archblocks now has mr support. mr repos won't authenticate yet, so you'll have to run "mr update" after logging back in if you have private repos being pulled down with mr, but as far as public repos go (public dotfiles, for instance), archblocks is now a complete start-to-finish install solution.

altercation · 2012-10-08 20:27:52

I've added a query option if you leave out the INSTALL_DRIVE value (or leave it blank). Additionally, the sample block for efi_gummiboot bootloader install now handles drives other than /dev/sda automatically (i.e. it uses the install drive as you define it in your config or as you select it interactively if that value has been left blank as mentioned).

The responsibility for setting the BOOT_DRIVE variable falls to the filesystem block (see the current efi-luks-gpt example filesystem block). As intended, the filesystem block is doing most of the heavy lifting in terms of system config.

altercation · 2012-10-10 03:00:00

Now supports pure systemd installation. INIT_MODE=systemd uses a pure systemd block. I could make this more configurable but since we're close to being pure systemd by default, I'm not making a lot of configuration options there.

altercation · 2012-10-16 21:18:53

Just merged dev branch into master. Added more prompts to filesystem setup sample block (new helper code to prompt for swap size), better systemd support, sensors, and other general improvements.

onny · 2013-01-27 18:46:21

Thank you for this! Do you know if it's also possible to autodeploy an ArchLinux installation using Vigrint/Chef/Puppet/VeeWee/etc. ?

szboardstretcher · 2013-05-05 20:20:08

So is there like a 30s quick-start guide to use this?

For example:

Burn iso to usb
use git to checkout from X://blah
copy files to /xx/ on usb
boot off of usb
go to directory
edit file x with your settings
run script
etc..

I dont see any documentation as to actually how to use it. Just a quick blurb about 'you have to configure a file to suit your installation' and really nothing else.

Thanks

pgoetz · 2015-07-17 21:39:58

The github for this is still available, but it looks like the code hasn't been touched in 3 years? I took a quick peak at some of the blocks and it seems that at least some of the stuff in the script I looked at is obsolete.

ewaller · 2015-07-17 21:57:16

Time to close this old thread

WorMzy · 2015-07-18 14:40:22

Closing, since ewaller forgot.

Arch Linux

#1 2012-09-25 19:58:03