APIC - idea about an ArchLinux configuration script repository

nlight · 2010-07-29 10:14:21

I've come up with an idea for a program that deals with system configuration automation. I've started to write something like a design doc and here it is.
I'm open to suggestions as if this would be useful to the community and if there is a point in developing it.

______________________________________
/ ArchLinux Post-Install Configuration \
\__________Design Document_____________/

1. What is APIC?

ArchLinux Post-Install Configuration (or APIC) is a database of user-uploaded and maintained scripts that help in the installation and configuration of your ArchLinux system.

2. What APIC is _not_?

APIC is not a package manager. It doesn't deal with packaging/installation, but uses pacman and the AUR to install packages.
APIC is also not a pacman front-end. It uses pacman to install packages, but all other functions are related to configuration, not installation.

2. How does it work?

APIC provides several modules that help it to do its purpose. Here is a brief description of each, a detailed description will follow further in the document.

A) APIC repository
The repository is a daemon program that runs on a central server. It hosts all uploaded scripts, supports versioning and comments. Scripts are divided into two categories
- Trusted scripts
- Untrusted scripts

Trusted scripts are the core of APIC. They are previously untrusted scripts which were voted for by the community or selected by APIC admins and then promoted to trusted.
Untrusted scripts are these uploaded by users. Due to the fact that the APIC client must be run under root makes them potentially dangerous.
The process and policies for a script to become trusted are detailed further in the document.

B) APIC client
The client is a console program that mimics the ArchLinux installer and is instantly familiar to anyone who has installed ArchLinux.
It uses the Dialog package to display information and receive user input.
On its first run it provides the user with a wizard, which collects information about the current state of the system.
It then updates its script database and provides a menu-based interface for selecting a script to run.

C) APIC script interpreter
The interpreter runs the APIC scripts.
It provides the built-in library functions and sets some useful variables.
It also deals with access control in a safe way. Full system access shall be granted to no script (;
The specific schema that APIC would use for this will be detailed further down.

3. What is an APIC script?

An APIC script is a small program that performs system configuration or maintanence. By definition, these scripts should be as simple as possible with many scripts sourcing others (otherwise we'd end up with a big mess of untested/unmaintained scripts that do no better than the forums or the wiki). Human-readability is also key. APIC scripts will be heavily sandboxed (that depends on the context of the particular script, of course. A script that relies on having direct harddisk access cannot really be sandboxed enough. For these particular cases where the script requires extraordinary control over your system, APIC would inform the user, display the part of the script that requires a higher level of access (so that the user can check it himself) and ask for his/her permission.)

The scripts shall be written in an interpreted language similar to bash (with some major restrictions to allow proper sandboxing). The syntax for this language will be defined in this document.
Each script is given its own temporary environment. While filesystem access is transparent,no real changes are made before the script finishes execution and asks for permission to modify system files, letting the user double-check the changes made. If permission is granted, APIC makes a backup of the original and only then replaces the actual files.

*** Example APIC script ***
This script installs and configures the Broadcom BCM43XX driver if the supported hardware is available on the system.
Comments are C++ style (//).

// Every script begins with the apic header
##!apic
// then some information about the script
// Note that it does not include version information. Versioning is done by the repository and the appropriate info is added to // the filename.

name = "Broadcom BCM43XX"
category = "drivers:wlan"
author = "Jane Doe"

// Each script must export two functions
// The first one is this:

function is_relevant() {
// This function shall determine if the script is appropriate to run on the system (dependencies met, hardware available)
// and return true or false accordingly. This function will therefore rely heavily on the built-in library
// for hardware detection and the pacman&AUR interface

if hw_detect_device("Broadcom Corporation BCM43*") {
return true
}

return false
}

// The second function is the script body – main()

function main() {
pkg_install_yaourt("broadcom-wl")

// here is an example of a huge security risk
// system() is unsafe so all calls shall be matched against a database of known safe commands
// in this particular case depmod -a would be marked as safe and the script would continue without interruption
// in the other scenario where the call is marked as unsafe, APIC would prompt the user for permission.
system("depmod -a")

rcconf_modules_add("lib80211_crypt_tkip wl")
rcconf_modules_blacklist("b43 ssb")

// end successfuly
return false
}

4. Built-in library

Here is a short overview of the first built-in functions.

(User interaction)
dialog_msgbox("text")
dialog_menu("title", "option1", "option2", "option3", ..)
dialog_inputbox("title", "options")
dialog_passwordbox("title", "options")
dialog_progressbar("title", "options")
dialog_progressbar_set(progress in %)

(Hardware detection)
hw_detect_device("device name or part of name")
hw_detect_usb_device("usb ID")
hw_load_modules("module1 module2 ..")
hw_unload_modules("module1 module2 ..")

(Configuration file parsers)
// rc.conf parser
rcconf_set_locale()
rcconf_modules_set_autoload(true/false)
rcconf_modules_add("module1 module2 ..")
rcconf_modules_blacklist("module1 module2 ..")
rcconf_use_lvm(true/false)
rcconf_hostname("name")
rcconf_daemons_add("daemon1 daemon2 ..")
rcconf_daemons_blacklist("daemon1 daemon2 ..")

// standard var=value conf file parser
conf_open_file("filename")
conf_close_file()
conf_get_value("var")
conf_set_value("var", "value")
conf_search_replace("search for", "replace with")
conf_diff("diff file")

// udev conf file parser
..
// xml parser
..

[WIP]

Last edited by nlight (2010-07-30 12:34:02)

karol · 2010-07-29 11:25:33

> 3. What is an APIC script?
> An APIC script is a set of subroutines that do a specific system-related task. Scripts are written in Python.
That's kinda limiting. Also, I don't think most _users_ speak decent python.

nlight · 2010-07-29 12:51:57

karol wrote:

> 3. What is an APIC script?
> An APIC script is a set of subroutines that do a specific system-related task. Scripts are written in Python.
That's kinda limiting. Also, I don't think most _users_ speak decent python.

I guess the library could be ported to almost anything. Python, Perl, Ruby and bash would be my first choices.
I'm thinking of writing it in C and using SWIG to generate bindings.

Last edited by nlight (2010-07-29 12:53:50)

Ari'osika · 2010-07-30 10:08:14

Me likes this idea although I have no idea on how this should be implemented. If we're talking a kind of implementation into the Arch installer, i don't what that'll entail, but if we're talking server-space I have / can setup some and use that as a script-repo.

(Also, I've prototyped a CLI-based perl script that pretty much does this).

Dieter@be · 2010-07-30 11:35:31

AIF offers you to integrate these dialogs, additional packages, and automated additional logic into the installation process, by just writing a simple bash file (called an aif procedure/library/config)

Aif however does not offer any kind of public repo to share such files. Although it has always been my goal that at some point such repository should exist.

Last edited by Dieter@be (2010-07-30 11:37:58)

nlight · 2010-07-30 11:43:01

Okay, I've significantly updated the current ideas. Take a new look at the first post.

afaik AIF is not really designed for this, but if it really does the same thing I'll let it go.
I'm still open to suggestions. Development won't really begin until this design doc is finished, let the ideas flow.

tomk · 2010-07-30 11:50:08

Obviously, I'm too old-school to understand the need for this. One of the purest joys of Arch for me is configuring it - myself, manually,

That said, if it scratches your itch, good luck with it.

Labello · 2010-07-30 11:59:30

tomk wrote:

Obviously, I'm too old-school to understand the need for this. One of the purest joys of Arch for me is configuring it - myself, manually.

so true!

bernarcher · 2010-07-30 12:16:42

I have mixed feelings about this. At first reading it sounded like a good idea. But then similar reservations as tomk's came to my mind. Arch systems are configured to one's individual needs. This is one of the main characteristics wich sets Arch apart from the main bulk of distributions. As tomk said, you can configure it to your heart's content.

On the other hand there are basic configuring needs. And of course examples of successful configuring to fulfil some given need. For these some kind of database of easily applicable scripts surely would be a good idea.

Yet we must remain careful not to try too much: keep it simple and stupid. That's the real Arch way.

Last edited by bernarcher (2010-07-30 12:17:37)

Dieter@be · 2010-07-30 12:48:01

nlight wrote:

Okay, I've significantly updated the current ideas. Take a new look at the first post.
afaik AIF is not really designed for this, but if it really does the same thing I'll let it go.
I'm still open to suggestions. Development won't really begin until this design doc is finished, let the ideas flow.

AIF is designed for exactly this. See http://github.com/Dieterbe/aif/blob/master/README

There are however some differences.
AIF currently has no repository component, although it does support downloading proceduces over http. (I wrote that with this use case in mind)
AIF also has no separation between trusted/untrusted scripts, it doesn't sandbox scripts and does not try to be smart and try to prevent you from executing certain commands. That is imho pointless overcomplication.
Users should only execute scripts which they trust (ie: after looking over them, like you can do with yaourt). simple as that.
I've always imagined an AUR-like repository of installation procedures/profiles for AIF, but never did it.

AIF comes with libraries that provide lowlevel features (see http://projects.archlinux.org/aif.git/tree/), for the UI library, I use http://github.com/Dieterbe/libui-sh/.
Also, all the features and interfaces presented to the end user are available as functions, which you can invoke from your own scripts.

na12 · 2010-07-30 12:57:19

tomk wrote:

Obviously, I'm too old-school to understand the need for this. One of the purest joys of Arch for me is configuring it - myself, manually,

That's true,but if you have to install and configure arch with limited time, this would be very helpful.

nlight · 2010-07-30 13:10:11

Dieter@be wrote:

AIF also has no separation between trusted/untrusted scripts, it doesn't sandbox scripts and does not try to be smart and try to prevent you from executing certain commands. That is imho pointless overcomplication.
Users should only execute scripts which they trust (ie: after looking over them, like you can do with yaourt). simple as that.

I respectfully disagree. When you've got a database (be it small or large) of user-uploaded scripts that must be run as root (AUR PKGBUILD's don't), a clear cut distinction between what you can completely trust and what not is mandatory. You can hide almost anything in a PKGBUILD if the script is complex enough. I believe security is key and "being smart" is a prerequisite for software such as this.
That being said, I believe the perfect middle-ground between simplicity and security exists and we can find it. More user response is needed to determine what the community wants/needs.

P.S. I actually ditched the idea of Python or Bash or Lua or Ruby scripts for the very same reason. Full-blown programming languages make things extremely easy to hide. A specialized language for system administration, such as the one proposed, that's easily human-readable and doesn't contain a way to hide what the script is actually doing would be nice.

Last edited by nlight (2010-07-30 13:14:45)

karol · 2010-07-30 13:23:45

> You can hide almost anything in a PKGBUILD if the script is complex enough
I don't sign what I don't understand, I don't invest in what I don't know.
May users use AUR so a malicious script is easier to spot.

I'm not into mass-deployment so I may miss the point. I install Arch once and run it forever :-)

nlight · 2010-07-30 14:21:41

karol wrote:

I don't sign what I don't understand, I don't invest in what I don't know.
May users use AUR so a malicious script is easier to spot.
I'm not into mass-deployment so I may miss the point. I install Arch once and run it forever :-)

Configuring your system from scratch is cool and all (I do it for all my computers), but you can't admit that the percentage of users that are advanced enough to do so is extremely limited. This kind of thinking will always lead users out, not in. I've been using Arch for the past 6 years and I love it so much I believe it's high time tools to ease its installation and configuration are developed (I'm aware of many cases of users switching from Arch, because they can't install wireless drivers or cannot get proper support for other hardware when drivers exist, but are too cumbersome to install. The wiki is also sparse in information on many devices found on the market and many users don't want to do a forum post about every issue). Afaik no simple tools such as this exist at the moment (at least in a workable state). It would be great if all users can benefit from the advanced configurations of others and I know writing a simple script to do so would be extremely easy for most of you guys (;

I'm really interested in how this goes. This is a really nice forum.

karol · 2010-07-30 14:28:52

nlight wrote:

karol wrote:
I don't sign what I don't understand, I don't invest in what I don't know.
May users use AUR so a malicious script is easier to spot.
I'm not into mass-deployment so I may miss the point. I install Arch once and run it forever :-)
Configuring your system from scratch is cool and all (I do it for all my computers), but you can't admit that the percentage of users that are advanced enough to do so is extremely limited. This kind of thinking will always lead users out, not in. I've been using Arch for the past 6 years and I love it so much I believe it's high time tools to ease its installation and configuration are developed (I'm aware of many cases of users switching from Arch, because they can't install wireless drivers or cannot get proper support for other hardware when drivers exist, but are too cumbersome to install. The wiki is also sparse in information on many devices found on the market and many users don't want to do a forum post about every issue). Afaik no simple tools such as this exist at the moment (at least in a workable state). It would be great if all users can benefit from the advanced configurations of others and I know writing a simple script to do so would be extremely easy for most of you guys (;
I'm really interested in how this goes. This is a really nice forum.

Ah, so it's about newbish users, now I more or less get it.
There's Chakra w/ a nice "sit back and watch" installer that automagically does everything for you.

nlight wrote:

The wiki is also sparse in information on many devices.

And you want to encode more info into your program? That's kinda impossible - not everything can be successfully "discovered" by a program.

nlight wrote:

Afaik no simple tools such as this exist at the moment (at least in a workable state).

http://chakra-project.org/tools.html

Last edited by karol (2010-07-30 14:30:26)

nlight · 2010-07-30 14:43:17

karol wrote:

Ah, so it's about newbish users, now I more or less get it.
There's Chakra w/ a nice "sit back and watch" installer that automagically does everything for you.

Chakra is more or less a Live CD. I'm not fully aware of its functions, but its extremely limited in options. APIC would not force anything on the user, just provide an easier way to do the more complicated stuff advanced users can.

karol wrote:

And you want to encode more info into your program? That's kinda impossible - not everything can be successfully "discovered" by a program.

That's the point of the is_relevant() function of the scripts. It would be fairly easy to code even complex checks for hardware configurations and settings using all kinds of external tools available (but not familiar to the not-so-advanced user).

Misfit138 · 2010-07-30 16:50:10

Community Contributions is a better home for this.....moving.

tomk · 2010-07-30 23:32:41

nlight wrote:

the percentage of users that are advanced enough to do so is extremely limited

Arch is intended for a particular subsection of linux users - this is stated clearly in the wiki. Of course, anyone is welcome to (try to) use Arch, but the kind of automation you are proposing is incompatible with Arch's principles.

nlight wrote:

I believe it's high time tools to ease its installation and configuration are developed

Why?

nlight wrote:

I'm aware of many cases of users switching from Arch, because they can't install wireless drivers or cannot get proper support for other hardware when drivers exist, but are too cumbersome to install

Is this a problem? I would be more inclined to see this as a natural progression for such users who, having tried Arch and found it unsuitable for their particular requirements, decide to move on to other distros that are a better match.

nlight wrote:

The wiki is also sparse in information on many devices found on the market

True, but only because Archers are accustomed to using all available resources, not just the Arch wiki. Anyone who finds the wiki lacking is welcome to add to it.

anonymous_user · 2010-07-30 23:59:32

tomk wrote:

nlight wrote:
the percentage of users that are advanced enough to do so is extremely limited
Arch is intended for a particular subsection of linux users - this is stated clearly in the wiki. Of course, anyone is welcome to (try to) use Arch, but the kind of automation you are proposing is incompatible with Arch's principles.

To be fair, its not like this automated solution will become the new default.

tomk · 2010-07-31 00:08:39

Of course not - I'm just answering nlight's point about non-advanced users.

Ari'osika · 2010-07-31 06:34:50

@nlight Got your email and am working on solution toward that (my server is being, well, my server ). But re-reading this topic; wouldn't it be better to just set up a custom repo/database/website containing nothing but the auto-config scripts?

In theory it would goes as follows;

pacman -S elinks
**navigates to repo, obtains info**
wget blahblah
tar -xvf blahblah.tar.gz
chmod +x blahblah
**execute** blahblah

Or am I misssing something? Which is probably true.

Personally and minus the elinks part, I use this method quite often in part because I reformat my drives every month or so.

nlight · 2010-07-31 09:26:44

anonymous_user wrote:

tomk wrote:
nlight wrote:
the percentage of users that are advanced enough to do so is extremely limited
Arch is intended for a particular subsection of linux users - this is stated clearly in the wiki. Of course, anyone is welcome to (try to) use Arch, but the kind of automation you are proposing is incompatible with Arch's principles.
To be fair, its not like this automated solution will become the new default.

Well, yeah. I'm not actually proposing that Arch should become Ubuntu-like, bundled with basically everything. That would be silly. It's perfect, because it's simple, right? (;
I'm not sure if I got my point around correctly. APIC should be a repository for sharing scripts, not an automated system configuration solution. The automation part would come by itself sooner or later if sufficiently advanced scripts are developed, but that's hardly the point. The only difference between posting some neat configuration script in the forums/wiki or apic would be that the apic client will restrict the damage a script can do to your system if used incorrectly or was malicious in the first place (plus automatic versioning and a nice toolkit).

I have started writing the specifications for the scripting language and have developed a prototype of the client/interpreter. I should have something working in a couple of days.

Ari'osika wrote:

@nlight Got your email and am working on solution toward that (my server is being, well, my server ). But re-reading this topic; wouldn't it be better to just set up a custom repo/database/website containing nothing but the auto-config scripts?

IMO the server part should consist of a webpage for uploading scripts and, yes, the latest nightly build of all uploaded scripts (also the latest version of the client).

Last edited by nlight (2010-07-31 09:58:52)

Ari'osika · 2010-08-02 09:44:26

Question. What would you need in terms of the server? Server-space, FTP account?

bharani · 2010-08-02 10:10:58

This is a good idea. IMO it should be like AUR. Just a bash script with lot of comments. Users checks the script and use it. And as arch users are advanced users they should be able to spot troubles.

Dieter@be · 2010-08-03 17:50:09

bharani wrote:

This is a good idea. IMO it should be like AUR. Just a bash script with lot of comments. Users checks the script and use it. And as arch users are advanced users they should be able to spot troubles.

I agree mostly. He is trying to solve a social problem ("Can I trust the code of this person") by technical means (writing something pretty complicated with sandboxing and preventing scripts from doing certain things). This problem should be fixed in a social way. Like the AUR does (code review by the community and voting).

Too counter the "scripts can be complicated and hide things well" argument: a script which is too complicated to be understood entirely by at least some people, should be avoided by everyone. In other words: scripts *should* be as simple as possible, that way they are easy to review, easy to use, easy to hack, and easy to check the trustworthyness.

Last edited by Dieter@be (2010-08-03 17:50:48)

Arch Linux

#1 2010-07-29 10:14:21

APIC - idea about an ArchLinux configuration script repository

#2 2010-07-29 11:25:33

Re: APIC - idea about an ArchLinux configuration script repository

#3 2010-07-29 12:51:57

Re: APIC - idea about an ArchLinux configuration script repository

#4 2010-07-30 10:08:14

Re: APIC - idea about an ArchLinux configuration script repository

#5 2010-07-30 11:35:31

Re: APIC - idea about an ArchLinux configuration script repository

#6 2010-07-30 11:43:01

Re: APIC - idea about an ArchLinux configuration script repository

#7 2010-07-30 11:50:08

Re: APIC - idea about an ArchLinux configuration script repository

#8 2010-07-30 11:59:30

Re: APIC - idea about an ArchLinux configuration script repository

#9 2010-07-30 12:16:42

Re: APIC - idea about an ArchLinux configuration script repository

#10 2010-07-30 12:48:01

Re: APIC - idea about an ArchLinux configuration script repository

#11 2010-07-30 12:57:19

Re: APIC - idea about an ArchLinux configuration script repository

#12 2010-07-30 13:10:11

Re: APIC - idea about an ArchLinux configuration script repository

#13 2010-07-30 13:23:45

Re: APIC - idea about an ArchLinux configuration script repository

#14 2010-07-30 14:21:41

Re: APIC - idea about an ArchLinux configuration script repository

#15 2010-07-30 14:28:52

Re: APIC - idea about an ArchLinux configuration script repository

#16 2010-07-30 14:43:17

Re: APIC - idea about an ArchLinux configuration script repository

#17 2010-07-30 16:50:10

Re: APIC - idea about an ArchLinux configuration script repository

#18 2010-07-30 23:32:41

Re: APIC - idea about an ArchLinux configuration script repository

#19 2010-07-30 23:59:32

Re: APIC - idea about an ArchLinux configuration script repository

#20 2010-07-31 00:08:39

Re: APIC - idea about an ArchLinux configuration script repository

#21 2010-07-31 06:34:50

Re: APIC - idea about an ArchLinux configuration script repository

#22 2010-07-31 09:26:44

Re: APIC - idea about an ArchLinux configuration script repository

#23 2010-08-02 09:44:26

Re: APIC - idea about an ArchLinux configuration script repository

#24 2010-08-02 10:10:58

Re: APIC - idea about an ArchLinux configuration script repository

#25 2010-08-03 17:50:09

Re: APIC - idea about an ArchLinux configuration script repository

Board footer