You are not logged in.

#1 2010-10-30 19:22:02

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

varch - Virtual machine builder for ArchLinux

I just finished the initial release of varch!

varch is a tool used to easily generate an ArchLinux virtual machine images.

The google-code homepage is:
http://code.google.com/p/varch/

The package is in the AUR:
http://aur.archlinux.org/packages.php?ID=42428

A brief wikipage on what varch is and how to use it:
https://wiki.archlinux.org/index.php/Varch

Varch is CRAZY easy to use, and is far more simple that other distribution's virtual machine builders.

New improved support for Libvirt in 0.8

New support for external kernels and a number of bug fixes in 0.8.2

Let me know what you think!

0.8.2 is out!

Last edited by Ryujin (2010-12-20 07:13:46)

Offline

#2 2010-10-30 20:12:16

sugardeath
Member
Registered: 2010-03-02
Posts: 82

Re: varch - Virtual machine builder for ArchLinux

Nice!  I look forward to when this will build for other VMs.  I don't have a processor with the vtx or svm flags (I know what I'm looking for when I upgrade, though), but I will keep an eye on this.

Offline

#3 2010-10-30 21:05:00

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

You are in luck, I only really realized that I should support other virtual machines when I was putting together this post, so I updated my code and I will have a new version out that supports non kvm soon, like within an hour.

Offline

#4 2010-10-30 21:13:46

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

ok, 0.7.0 is out and there is a package on the AUR, this supports "generic" virtual machines for more than just kvm smile

Offline

#5 2010-10-30 22:26:40

sugardeath
Member
Registered: 2010-03-02
Posts: 82

Re: varch - Virtual machine builder for ArchLinux

Fast!  Nice!  I'll check it out tonight when I get home.

Edit:
OK, played around with it a bit.  Makes it really easy.  Now I've gotta learn AIF =P  Using this to generate a raw, generic image, I could then deploy that image on any machine, right?  I'm assuming so since I was able to get qemu to run a raw image of my netbook.

Also:  The generic image isn't recognized by Virtualbox.  I get the following error when trying to add the image:

Failed to open the hard disk /home/satoshi/arch.qcow2.
Could not get the storage format of the medium '/home/satoshi/arch64.qcow2' (VERR_NOT_SUPPORTED).

Details:
Result Code: 
VBOX_E_IPRT_ERROR (0x80BB0005)
Component: 
Medium
Interface: 
IMedium {1d578f43-5ef1-4415-b556-7592d3ccdc8f}
Callee: 
IVirtualBox {3f36e024-7fed-4f20-a02c-9158a82b44e6}

I realize that says qcow2, but I get that with both qcow2 and raw images (I did not expect the qcow2 image to work, but this was just the error I had up at the time).

Otherwise, they seem to work perfectly in qemu.

I like it!

Edit:  Also, when it says "Package installation will begin now.  You can watch the output in the progress window. Please be patient.", there is no progress window.  Is there a dependency I'm missing and it's not checking for?

Last edited by sugardeath (2010-10-31 10:23:56)

Offline

#6 2010-10-31 16:27:49

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

When it is installing packages it is inside aif, normally when you use aif to install a hardware install of arch a window is displayed with the output; but here there is no output window, sorry smile

As for virtualbox, I will have to take a look.

Offline

#7 2010-10-31 18:57:00

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: varch - Virtual machine builder for ArchLinux

Didn't check the code properly, but on first sight this looks cool.
And +1 for using aif, of course smile


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#8 2010-11-01 02:47:57

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Thanks Dieter! If you have any suggestions on my code I am all ears, it is all in python, python 3 of course.

As for aif, I have the utmost compliments, the simplicity and flexibility of aif and Arch has made this project very clean and straightforward!

As for the virtualbox issue, I added vdi as another supported format since I know that there are a lot of Archers that use virtualbox. So to make a virtualbox image:

# varch -f vdi

you don't need to declare the image as "generic" since it will assume as much for a virtualbox image.

I am excited to see that a few people are using this, if you find bugs please let me know, and if you have feature requests please put them here or on the google code page.

Offline

#9 2010-11-01 02:49:11

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Sorry vdi support is in 0.7.1, available on the AUR.

Offline

#10 2010-11-09 06:09:41

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Varch 0.7.8 is out! I did a lot of cleaning up in my code and made the output a lot cleaner, I put in checks for lvm so that aif will not destroy any existing partition. Also added a new aif file with a simple lvm setup.

Please test varch and tell me what you think or what features you want.

BTW, if you have suggestions or feature requests I would love to hear them!

Last edited by Ryujin (2010-11-09 06:14:10)

Offline

#11 2010-11-09 22:45:41

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: varch - Virtual machine builder for ArchLinux

I'm trying varch.

when it was in the creating FS step i freaked out a bit because my pc became inresponsive during the mkfs.ext4 calls, but luckily it was correctly working on the /dev/mapper/loop* files.
I checked /tmp/working.aif and it looks ok smile

However something else went wrong:

sudo varch
Password: 
############################################################################
#                    Creating virtual machine image                        #
############################################################################
#Checking that the aif configuration will be safe for the underlying system#
############################################################################
#                             Executing AIF                                #
############################################################################
Welcome to Arch Linux Installation Framework
Automatic procedure running profile /tmp/working.aif ...
Note that despite our careful coding and proper testing there may still be bugs in this software.

When you are doing this installation on a system where some data must be preserved, we suggest you make a backup first
Partitioning /dev/mapper/loop0
Disabling all swapspace...
Phase 1: Creating filesystems & blockdevices
Making ext2 filesystem on /dev/mapper/loop0p1
Making swap filesystem on /dev/mapper/loop0p2
Making ext4 filesystem on /dev/mapper/loop0p3
Phase 2: Mounting filesystems
Mounting /dev/mapper/loop0p3 (ext4) on /
Mounting /dev/mapper/loop0p1 (ext2) on /boot
Swaponning /dev/mapper/loop0p2
Done processing filesystems/blockdevices
Partitions and filesystems made successfully
Refreshing package database...
Package installation will begin now.  You can watch the output in the progress window. Please be patient.
Installing... Please Wait
WARNING: Installation Failed (see errors below)
error: failed to prepare transaction (package architecture is not valid)
:: package attr-2.4.44-1-x86_64 does not have a valid architecture
:: package bash-4.1.009-1-x86_64 does not have a valid architecture
:: package binutils-2.20.1-4-x86_64 does not have a valid architecture
:: package bzip2-1.0.6-1-x86_64 does not have a valid architecture
:: package coreutils-8.6-1-x86_64 does not have a valid architecture
:: package cryptsetup-1.1.3-1-x86_64 does not have a valid architecture
:: package dash-0.5.6.1-1-x86_64 does not have a valid architecture
:: package dcron-4.4-2-x86_64 does not have a valid architecture
:: package device-mapper-2.02.75-1-x86_64 does not have a valid architecture
:: package dhcpcd-5.2.8-1-x86_64 does not have a valid architecture
:: package diffutils-3.0-1-x86_64 does not have a valid architecture
:: package e2fsprogs-1.41.12-1-x86_64 does not have a valid architecture
:: package file-5.04-2-x86_64 does not have a valid architecture
:: package findutils-4.4.2-2-x86_64 does not have a valid architecture
:: package gawk-3.1.8-1-x86_64 does not have a valid architecture
:: package gcc-libs-4.5.1-1-x86_64 does not have a valid architecture
:: package gen-init-cpio-2.6.32-1-x86_64 does not have a valid architecture
:: package gettext-0.18.1.1-1-x86_64 does not have a valid architecture
:: package glibc-2.12.1-3-x86_64 does not have a valid architecture
:: package grep-2.7-1-x86_64 does not have a valid architecture
:: package grub-0.97-17-x86_64 does not have a valid architecture
:: package gzip-1.4-1-x86_64 does not have a valid architecture
:: package initscripts-2010.07-1-x86_64 does not have a valid architecture
:: package iputils-20100214-2-x86_64 does not have a valid architecture
:: package jfsutils-1.1.14-1-x86_64 does not have a valid architecture
:: package kernel26-2.6.35.8-1-x86_64 does not have a valid architecture
:: package less-436-1-x86_64 does not have a valid architecture
:: package libusb-0.1.12-4-x86_64 does not have a valid architecture
:: package logrotate-3.7.9-1-x86_64 does not have a valid architecture
:: package lvm2-2.02.75-1-x86_64 does not have a valid architecture
:: package lzo2-2.03-1-x86_64 does not have a valid architecture
:: package mailx-8.1.1-7-x86_64 does not have a valid architecture
:: package man-db-2.5.7-1-x86_64 does not have a valid architecture
:: package mdadm-3.1.4-1-x86_64 does not have a valid architecture
:: package net-tools-1.60-14-x86_64 does not have a valid architecture
:: package pacman-3.4.1-1-x86_64 does not have a valid architecture
:: package pciutils-3.1.7-1-x86_64 does not have a valid architecture
:: package pcmciautils-017-1-x86_64 does not have a valid architecture
:: package perl-5.12.1-3-x86_64 does not have a valid architecture
:: package ppp-2.4.5-1-x86_64 does not have a valid architecture
:: package procps-3.2.8-1-x86_64 does not have a valid architecture
:: package psmisc-22.13-1-x86_64 does not have a valid architecture
:: package rp-pppoe-3.10-5-x86_64 does not have a valid architecture
:: package sed-4.2.1-2-x86_64 does not have a valid architecture
:: package shadow-4.1.4.2-3-x86_64 does not have a valid architecture
:: package sysfsutils-2.1.0-5-x86_64 does not have a valid architecture
:: package syslog-ng-3.1.2-1-x86_64 does not have a valid architecture
:: package sysvinit-2.88-2-x86_64 does not have a valid architecture
:: package tar-1.23-3-x86_64 does not have a valid architecture
:: package tcp_wrappers-7.6-12-x86_64 does not have a valid architecture
:: package texinfo-4.13a-4-x86_64 does not have a valid architecture
:: package udev-163-1-x86_64 does not have a valid architecture
:: package usbutils-0.90-1-x86_64 does not have a valid architecture
:: package util-linux-ng-2.18-3-x86_64 does not have a valid architecture
:: package vi-050325-3-x86_64 does not have a valid architecture
:: package wget-1.12-2-x86_64 does not have a valid architecture
:: package which-2.20-3-x86_64 does not have a valid architecture
:: package wpa_supplicant-0.7.3-1-x86_64 does not have a valid architecture
:: package xfsprogs-3.1.3-1-x86_64 does not have a valid architecture
:: package openssh-5.6p1-1-x86_64 does not have a valid architecture

Package Installation FAILED.
sed: can't read /tmp/tmpYuFtcK/etc/fstab: No such file or directory
/usr/lib/aif/core/libs/lib-blockdevices-filesystems.sh: line 332: /tmp/tmpYuFtcK/etc/fstab: No such file or directory
grep: /tmp/tmpYuFtcK/etc/rc.conf: No such file or directory
Generating glibc base locales...
chroot: failed to run command `locale-gen': No such file or directory
sed: can't read /tmp/tmpYuFtcK/etc/rc.conf: No such file or directory
awk: fatal: cannot open file `/tmp/tmpYuFtcK//etc/pacman.d/mirrorlist' for reading (No such file or directory)
/usr/lib/aif/core/libs/lib-ui-interactive.sh: line 27: /tmp/tmpYuFtcK//etc/pacman.d/mirrorlist: No such file or directory
sed: can't read /tmp/tmpYuFtcK/etc/rc.conf: No such file or directory
grep: /tmp/tmpYuFtcK/etc/mkinitcpio.conf: No such file or directory
sed: can't read /tmp/tmpYuFtcK/etc/mkinitcpio.conf: No such file or directory
Rebuilding initcpio images ...
WARNING: MKINITCPIO FAILED - SYSTEM MAY NOT BOOT
chroot: failed to run command `/sbin/mkinitcpio': No such file or directory
Generating glibc base locales...
chroot: failed to run command `locale-gen': No such file or directory
Generating GRUB device map...
This could take a while.

 Please be patient.
Probing devices to guess BIOS drives. This may take a long time.
/dev/loop0 does not have any corresponding BIOS drive.
grep: /tmp/aif/aif-dev.map: No such file or directory
/usr/lib/aif/core/libs/lib-blockdevices-filesystems.sh: line 289: per/loop0p1-1: division by 0 (error token is "-1")
Your root boot device could not be autodetected by setup.  Ensure you adjust the 'root (hd0,0)' line in your GRUB config accordingly.
sed: can't read /tmp/tmpYuFtcK/boot/grub/menu.lst: No such file or directory
/usr/lib/aif/core/libs/lib-ui-interactive.sh: line 1144: /tmp/tmpYuFtcK/boot/grub/menu.lst: No such file or directory
Execution Report:
-----------------
Phase preparation: Success
Phase basics: Success
Phase system: Success
Phase finish: Success
############################################################################
#                        Setting up the bootloader                         #
############################################################################
############################################################################
#                     Cleaning up the build environment                    #
############################################################################
Unmounting build environment
restoring swap state

< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#12 2010-11-10 07:13:40

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Sweet, problems! Thanks Dieter for testing this, since I am a one man show smile
As for getting freaked out, you are right to do so since you understand aif, I do go through a number of checks to make sure that nothing in the working.aif file will be in your running system, and so far I have been unable to make varch destroy one of my systems. The reason it takes so long is that I have an evil hack, you see vmbuilder and thincust basically recreate the installers, because tools like anaconda are not KISS, so they execute parted on the image file and then run kpartx.
The problem I run into with aif is that it does the partitioning, so I need to run kpartx in a loop in the background while aif starts. This makes it possible for aif to see the partitions it creates on the loopback device, but since kpartx is running a billion times a second in the background it slown down mkfs smile I haven't quite figured out a better way yet, I am working on it.

As for your problem, it obviously did not install, and looks like an arch problem, is your system 32 or 64 bit? Honestly I only have 64 bit systems which all have the virtualization extensions. If it is a 32 bit system open up the /etc/varch/base.aif and change the syncurl to use a 32 bit repo.

Please let me know and thanks again for testing.

BTW, I assume it worked before? Or is this your first try?

Last edited by Ryujin (2010-11-10 07:15:33)

Offline

#13 2010-11-10 07:44:52

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Ok, it is an arch issue, for now you can change the arch in the aif file, I will look into making varch capable of building a vm for an arch other than the running system.

Offline

#14 2010-11-10 08:01:43

sugardeath
Member
Registered: 2010-03-02
Posts: 82

Re: varch - Virtual machine builder for ArchLinux

edit: nevermind, didn't read your post fully

Last edited by sugardeath (2010-11-10 08:02:29)

Offline

#15 2010-11-10 08:47:24

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: varch - Virtual machine builder for ArchLinux

was my first try.
changing arch to i686 fixed the problem.

Furthermore:
You currently check for root before checking argument flags.  I should be able to run '-h or '--help' as any user.

You should keep in mind AIF is not just a tool, it's a framework.  Read up about phases and workers in /usr/share/aif/docs/README
What you need to do in this case, is look at /usr/lib/aif/core/procedures/automatic, copy the worker_prepare_disks() function and override it in your aif files.
Between the process_disks and process_filesystems calls you can put your kpartx logic.
Since you will be doing this in all cases (in both base.aif, base-lvm.aif and maybe others) you could put your "common overrides" in a separate bash file (/usr/lib/varch/aif-overrides.sh or something) and source it from base.aif and base-lvm.aif.

Also I saw varch has code for grub, why? Why not just let aif take care of grub?


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#16 2010-11-10 16:19:19

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Ah, thank you! That will help considerably.
As for grub, I have had trouble getting aif to install it on the loopback device, I think I need to create a device map for it. I will add it to the issues I need to solve.

Offline

#17 2010-11-10 22:28:28

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: varch - Virtual machine builder for ArchLinux

Very nice work, I will have to test this with ArchServer big_smile

Offline

#18 2010-11-10 22:30:35

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Yes! I suspect it will work just fine. Please let me know if I need to make any changes for it to work with ArchServer.

Offline

#19 2010-11-10 22:34:27

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

BTW, does ArchServer have an AUR for me to post a package? Or do you just use the mainline AUR?

Offline

#20 2010-11-11 10:55:17

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: varch - Virtual machine builder for ArchLinux

Ryujin wrote:

BTW, does ArchServer have an AUR for me to post a package? Or do you just use the mainline AUR?

We're just "piggy backing" on the AL AUR at the moment... Everything should be compatible, so no point duplicating and spreading out a good thing wink

Offline

#21 2010-11-14 07:12:12

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

0.7.9 is out
I have fixed a number of small bugs and introduced a few new features.
Overlay support is final - You can create a directory structure of files that are copied into the virtual machine. This is nice because, while aif is great at modifying the system inline, if you want to replace a BUNCH of files an overlay will be easier -o|--overlay
Support for auto generation of a script to start the vm in kvm -K|--kvm-conf

Introduced a bunch of new aif files and fixed some problems in the partition handling

Tell me what you think!

Offline

#22 2010-11-14 11:50:09

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: varch - Virtual machine builder for ArchLinux

Ryujin wrote:

A numer of operations may need to be executed - post install - like the
application overlay and automated instalation of AUR packages.

You can install AUR packages with AIF. put something like yaourt or clyde in RUNTIME_PACKAGES (and make it available) and then you can put AUR packages in TARGET_PACKAGES
Of course if you want to use plain wget && makepkg && pacman -U then it's probably easier to do it after AIF.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#23 2010-11-16 00:33:07

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

Ahh, I need to look  into exactly what RUNTIME_PACKAGES and RUNTIME_REPOSITORIES do, but that solves the problem I was having with putting them in the main sequence.

And wow, you are reading my source code - this is my first real open source project, I have submitted many patches before but never done something like this from scratch, it is an interesting feeling knowing that someone else is actually reading my code! Thank you very much for your help, I am amazed how much easier it is to write open with positive suggestions coming in and good communication(I have spend many years in a corporate code farm).

Offline

#24 2010-11-16 00:43:57

Ryujin
Forum Fellow
From: Centerville, Utah
Registered: 2005-05-12
Posts: 246
Website

Re: varch - Virtual machine builder for ArchLinux

So, if I want to use yaourt in AIF, I add yaourt to runtime packages, and I need to make a binary yaourt package (and deps) available in a repository listed in RUNTIME_REPOSITORIES? I can't just have it installed on the host system?

Offline

#25 2010-11-17 09:57:21

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: varch - Virtual machine builder for ArchLinux

Ryujin wrote:

So, if I want to use yaourt in AIF, I add yaourt to runtime packages, and I need to make a binary yaourt package (and deps) available in a repository listed in RUNTIME_REPOSITORIES? I can't just have it installed on the host system?

both approaches should work.  I'm always thinking about the case where users boot our livecd and run aif in there, but in your case you can of course just install stuff in the users' system.

however, now that i think about it there are two issues:
- yaourt does not support a --config flag (even though the author claims yaourt supports the same arguments as pacman), so you should probably use clyde instead
- you need a way to make aif use clyde instead of pacman: you will need to override PACMAN_TARGET in your config file


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

Board footer

Powered by FluxBB