New project: AIF-NG

bsaner · 2017-05-07 19:27:50

(as posted to the arch-general mailing list...)

______________________
Hello!

Some of you may remember AIF (Arch Installation Framework). However, the project is dead and has been for quite some time.

Being that most major distributions have a way of installing their distribution in an unattended manner/via PXE:

-Debian (and derivatives, incl. Ubuntu): preseed
-RHEL/CentOS/SuSE (and derivatives): kickstart (cobbler, etc.)

I felt that Arch would benefit from something similar. As such, I have written AIF-NG. It's written in python 3 and uses an XML-driven configuration (see the documentation for details).

I would welcome feedback on this (preferably via my bugtracker unless it happens to be discussion-oriented), as I'd like to get it mature enough to propose it for inclusion in the official Arch install media (currently for this to be supported, that would only require two things: the "python" and "aif" (or "aif-git") packages, which can be found on the AUR).

______________________

Again, I want to stress- please please please use the bugtracker, do NOT post bug reports in here. But other than that, do y'all have any thoughts?

R00KIE · 2017-05-07 19:38:40

Are you planning to create a tool to help create the xml?

bsaner · 2017-05-07 19:42:19

yep, it's in the TODO! i'm not sure what would be the best approach for it, though.

Alad · 2017-05-07 20:05:22

Note there's an "official" installer in the form of ArchBoot. If the plan is to upstream this, you might want to look into coordinating efforts or get some more direct feedback.

https://www.archlinux.org/packages/extra/any/archboot/

bsaner · 2017-05-07 20:06:36

yep- "as I'd like to get it mature enough to propose it for inclusion in the official Arch install media"

edit:
did you mean archiso, not archboot?

lftp mirror.us.leaseweb.net:/archlinux/iso> ls latest
drwxr-xr-x 4 ftp ftp 4096 May 01 07:04 arch
-rw-r--r-- 1 ftp ftp 504365056 May 01 07:05 archlinux-2017.05.01-x86_64.iso
-rw-r--r-- 1 ftp ftp 310 May 01 07:12 archlinux-2017.05.01-x86_64.iso.sig
-rw-r--r-- 1 ftp ftp 33568 May 01 07:13 archlinux-2017.05.01-x86_64.iso.torrent
-rw-r--r-- 1 ftp ftp 140045137 May 01 07:11 archlinux-bootstrap-2017.05.01-x86_64.tar.gz
-rw-r--r-- 1 ftp ftp 310 May 01 07:12 archlinux-bootstrap-2017.05.01-x86_64.tar.gz.sig
-rw-r--r-- 1 ftp ftp 145 May 01 07:13 md5sums.txt
-rw-r--r-- 1 ftp ftp 161 May 01 07:13 sha1sums.txt
lftp mirror.us.leaseweb.net:/archlinux/iso> ls archboot/latest
-rw-r--r-- 1 ftp ftp 468713472 Dec 09 23:38 archlinux-2016.12-1-archboot-network.iso
-rw-r--r-- 1 ftp ftp 33404 Dec 09 23:38 archlinux-2016.12-1-archboot-network.iso.torrent
-rw-r--r-- 1 ftp ftp 1280491520 Dec 09 23:06 archlinux-2016.12-1-archboot.iso
-rw-r--r-- 1 ftp ftp 64377 Dec 09 23:38 archlinux-2016.12-1-archboot.iso.torrent
drwxr-xr-x 2 ftp ftp 4096 Dec 09 23:58 boot
-rw-r--r-- 1 ftp ftp 300 Dec 09 23:38 md5sum.txt

Last edited by bsaner (2017-05-07 22:06:59)

R00KIE · 2017-05-07 22:06:21

bsaner wrote:

yep, it's in the TODO! i'm not sure what would be the best approach for it, though.

That's good as I expect most people (me included) would steer away from it unless they are really motivated or it's their last resource. The reason is that xml is not human friendly when it comes to editing, specially when things start to have some complexity.

As to how to approach it, no idea but you probably want to get that tool working sooner rather than later. It will probably help everyone including you, I suppose it will be easier to handle bugs reports with "I input this and that in the interface and it doesn't work" rather than "I have this beautiful hand crafted xml and it doesn't work" and upon further inspection it turns out to be a pebcak making you waste time. Then again I'm not a developer and I don't handle bug reports so I can be completely off the mark here, you might want to seek guidance from others with experience with these things.

bsaner · 2017-05-07 22:11:19

yeah; i was moreso thinking along the lines of "how do i want it to prompt for input". i can probably have something put together by the end of next week. i might make it depend on python-lxml, though; the stdlib xml module's a little janky.

Last edited by bsaner (2017-05-07 22:11:46)

bsaner · 2017-05-11 22:38:46

R00KIE-

how does this look? it might be simplistic, but the XML config itself really isn't that complex and I include one or two sample configs.

[2017-05-11 18:27:34.260475] Beginning configuration...
You may reply with 'wikihelp' on the first prompt of a question for the relevant link(s) in the Arch wiki (and other resources).


What disk(s) would you like to be configured on the target system?
	If you have multiple disks, separate with a comma (e.g. '/dev/sda,/dev/sdb'): /dev/sda

Configuring disk /dev/sda ...
* What format should this disk use (gpt/bios)? gpt
* How many partitions should this disk have? (Maximum: 256) 2
** Where should partition 1 start? Can be percentage [n%] or size [(+/-)n(K/M/G/T/P)]: 0%
** Where should partition 1 stop? Can be percentage [n%] or size [(+/-)n(K/M/G/T/P)]: 90%
** What filesystem type should partition 1 be? See wikihelp for valid fstypes: 8300
	Selected Linux filesystem
** Where should partition 2 start? Can be percentage [n%] or size [(+/-)n(K/M/G/T/P)]: 90%
** Where should partition 2 stop? Can be percentage [n%] or size [(+/-)n(K/M/G/T/P)]: 100%
** What filesystem type should partition 2 be? See wikihelp for valid fstypes: ef00
	Selected EFI System

What mountpoint(s) would you like to be configured on the target system?
	If you have multiple mountpoints, separate with a comma (e.g. '/mnt/aif,/mnt/aif/boot').
	(NOTE: Can be 'swap' for swapspace.): /mnt/aif,/mnt/aif/boot

Configuring mountpoint /mnt/aif ...
* What device/partition should be mounted here? /dev/sda1
* What order should this mount occur in relation to others?
	Must be a unique integer (lower numbers mount before higher numbers): 1
* What filesystem type should this be mounted as (i.e. mount's -t option)? This is optional,
	but may be required for more exotic filesystem types. If you don't have to specify one,
	just leave this blank: 
* What, if any, mount option(s) (mount's -o option) do you require? (Multiple options should be separated
	with a comma). If none, leave this blank: 

Configuring mountpoint /mnt/aif/boot ...
* What device/partition should be mounted here? /dev/sda2
* What order should this mount occur in relation to others?
	Must be a unique integer (lower numbers mount before higher numbers): 2
* What filesystem type should this be mounted as (i.e. mount's -t option)? This is optional,
	but may be required for more exotic filesystem types. If you don't have to specify one,
	just leave this blank: 
* What, if any, mount option(s) (mount's -o option) do you require? (Multiple options should be separated
	with a comma). If none, leave this blank: 

Now, let's configure the network. Note that at this time, wireless/more exotic networking is not supported by AIF-NG.

What should the newly-installed system's hostname be?
	It must be in FQDN format, but can be a non-existent domain: aiftest.square-r00t.net

Please enter the name of the interface you would like to use.
Can instead be 'auto' for automatic configuration of the first found interface
with an active link. (You can only specify one auto device per system, and all subsequent
interface entries will be ignored.)

Interface device: auto
* Address for auto in CIDR format (can be an IPv4 or IPv6 address); use'auto' for DHCP/DHCPv6): auto
* Would you like 'ipv4', 'ipv6', or 'both' to be auto-configured? ipv4
Would you like to add more interfaces? ((y)es/(N)O) 

Now let's configure some basic system settings.
* What timezone should the newly installed system use? (Default is UTC): 
* What locale should the new system use? (Default is en_US.UTF-8): 
* What chroot path should the host use? This should be one of the mounts you specified above: /mnt/aif
* What keyboard layout should the newly installed system use? (Default is US): 
* Would you like to reboot the host system after installation completes? ((Y)ES/(n)o):

JohnBobSmith · 2017-05-12 00:07:59

Why do people think that Arch needs an installer? I'm willing to maintain an open mind about the subject, but in my case, I don't see the point of the installer. You miss a valuable learning opportunity to tinker/learn if you use an automated installer. In addition, you have the potential to end up with a broken system in many cases. This is why, AFAIK, random install scripts/etc are not supported. I just recently had to install Arch from scratch. The process really isn't that difficult, because the wiki is excellent. If you can read, use google (or man pages), and take the required time (15 minutes dependent on internet connection), you can install Arch. The nail in the coffin is that once installed, you should never ever have to re-install Arch from scratch. In my case I needed a re-install because I was migrating to an SSD. So simply put, I don't understand why Arch would need or should have an installer. Maybe someone can enlighten me?

bsaner · 2017-05-12 00:32:15

What is AIF-NG?

AIF-NG (Arch Installation Framework, Next Generation) is a means to automatically install Arch Linux. Think of it as something akin to Kickstart.

What it’s Not

AIF-NG is not intended to be a complete turnup solution. Instead, it’s useful to build up from baremetal and configure a system to a point where you can use another management tool (such as Ansible, Chef, Puppet, SaltStack, and others).
Though if you’re really gung-ho about it, I suppose you could use the post-script feature to fully turn up a box.
It is also not a magic bullet. It will not make an Arch Linux installation easier, nor is it designed to do that. Don’t file bug reports for this. It’s designed to make it faster. I recommend you follow the manual installation process several times first so you’re comfortable with the process and understand what’s happening behind the scenes. (If you find it too hard to understand, you may be interested in Antergos instead.)

nobody's forcing you to use it. nobody WOULD be forcing you to use it even if it were in the official ISO.

JohnBobSmith wrote:

You miss a valuable learning opportunity to tinker/learn if you use an automated installer.

that's what LFS is for. which i've done around seven times, now. or gentoo, to a lesser extent, which i ran for something like 12 years. pacstrap IS an install script. arch-chroot IS an install script.

JohnBobSmith wrote:

In addition, you have the potential to end up with a broken system in many cases.

this isn't some cockamamie half-assed GUI installer. it will either install or not. it has sanity checking. it's intended for unattended installs, like over PXE, to VMs, or the like. it's a way to take a machine with no OS/an undesirable OS on it, and say "you there; you are now a basic install of Arch Linux. install a minimal install and give me SSH into the actual install so i can finish it the way i like." (or, if you like, you can implement a hell of a lot of it in post scripts.)

the output you see above is not the tool. the output above is from a separate tool i'm working on, at R00KIE's request, to help generate a configuration file FOR AIF-NG. it is not AIF-NG itself.

Last edited by bsaner (2017-05-12 01:01:09)

bsaner · 2017-05-12 00:33:03

(oops, just noticed the duplicate "instead"; fixed)

Last edited by bsaner (2017-05-12 01:27:43)

Trilby · 2017-05-12 00:33:46

JohnBobSmith wrote:

You miss a valuable learning opportunity to tinker/learn if you use an automated installer.

Agreed - however, one could argue that every convenience in life is a missed opportunity. A good tractor is a missed opportunity to gain physical fitness from absurd levels of manual labor.

JohnBobSmith wrote:

In addition, you have the potential to end up with a broken system in many cases.

This does not seem to be a fair point. This is a downfall of many of the currently existing poorly made and unmaintained install scripts (or distrolets) - but this is not a necessary outcome of an installation tool. Arch did previously have an installation tool (AIF) which was discontinued because no one wanted to maintain it.

JohnBobSmith wrote:

This is why, AFAIK, random install scripts/etc are not supported.

Yes, random tools are not supported. But, as above, that does not preclude the possibility of there being a supported tool.

JohnBobSmith wrote:

I don't understand why Arch would need or should have an installer.

Despite my critiques of your reasoning above, I share this skepticism. I don't see your points against this project as carrying much weight - but I also do not see a need for such a tool.

Alad · 2017-05-12 09:18:55

did you mean archiso, not archboot?

Nope - it's a separate ISO. Though at closer inspection the aims are not necessarily the same.

Despite my critiques of your reasoning above, I share this skepticism. I don't see your points against this project as carrying much weight - but I also do not see a need for such a tool.

As I understand it, this is not so much an installer in the classical "I have no idea what I'm doing so I'll sacrifice a goat and pray this installer thing gives me a functional system" sense - as widely seen on the Internet - but rather a devops thing that allows you to quickly deploy a configured Arch install to a large amount of machines. There's some ways of achieving that already, like using custom packages and a modified archiso, but I don't think it's already solved in full generality, and this project might change that.

Last edited by Alad (2017-05-12 09:23:14)

bsaner · 2017-05-12 13:18:14

Alad wrote:

As I understand it, this is not so much an installer in the classical "I have no idea what I'm doing so I'll sacrifice a goat and pray this installer thing gives me a functional system" sense - as widely seen on the Internet - but rather a devops thing that allows you to quickly deploy a configured Arch install to a large amount of machines.

while i can't help but die a little inside at the word devops you've hit it on the button. configure once, run multiple times. i plan on implementing a couple extra things in the next coming releases once i have the core functionality stabilized and well-tested that would let you do things like have different configs per MAC address of a machine, kind of like how SIP endpoint device provisioning works (it first searches for a config matching the MAC address, and if it doesn't find it, checks for other model-specific configs, and eventually if it doesn't find any of those it uses a generic config). iPXE already supports this, too, but it'd be nice to support it right in the "client"/"install agent"

bsaner · 2017-05-15 17:07:08

R00KIE-

I've completed a configuration generator.

Install the aif or aif-git package from AUR, and run:

aif-config create -f /path/to/where/the/config/should/save/to.xml

(it should save it as "./aif.xml" if no -f <file> is specified, if you prefer it in the current working directory)

it'll run you through some prompts. if you don't know how to answer any of them, just reply with the answer wikihelp and it should give you some links to relevant documentation.

generate a couple configs and see how it works for you. it might take a couple tries- it errors out fatally if invalid input is given. but the prompts are pretty clear as to what's considered valid input.

Last edited by bsaner (2017-05-15 17:07:31)

bsaner · 2017-05-16 15:25:54

Alad- by the way, saw your wiki redirects- thanks! wasn't sure how to handle it; i haven't done much arch wiki editing.

bsaner · 2017-05-16 19:32:12

i've added some documentation to the arch wiki, for both the core/usage and for configuration.

Arch Linux

#1 2017-05-07 19:27:50

New project: AIF-NG

#2 2017-05-07 19:38:40

Re: New project: AIF-NG

#3 2017-05-07 19:42:19

Re: New project: AIF-NG

#4 2017-05-07 20:05:22

Re: New project: AIF-NG

#5 2017-05-07 20:06:36

Re: New project: AIF-NG

#6 2017-05-07 22:06:21

Re: New project: AIF-NG

#7 2017-05-07 22:11:19

Re: New project: AIF-NG

#8 2017-05-11 22:38:46

Re: New project: AIF-NG

#9 2017-05-12 00:07:59

Re: New project: AIF-NG

#10 2017-05-12 00:32:15

Re: New project: AIF-NG

#11 2017-05-12 00:33:03

Re: New project: AIF-NG

#12 2017-05-12 00:33:46

Re: New project: AIF-NG

#13 2017-05-12 09:18:55

Re: New project: AIF-NG

#14 2017-05-12 13:18:14

Re: New project: AIF-NG

#15 2017-05-15 17:07:08

Re: New project: AIF-NG

#16 2017-05-16 15:25:54

Re: New project: AIF-NG

#17 2017-05-16 19:32:12

Re: New project: AIF-NG

Board footer