You are not logged in.

#1 2005-09-07 06:49:32

butters
Member
Registered: 2005-08-31
Posts: 35

archbootstrap 0.4 with tarbuild

This is a continuation of a previous thread:

http://bbs.archlinux.org/viewtopic.php?t=4020

In short, archbootstrap lets you install a base archlinux system to the target directory of your choice from within any remotely standard *nix system, including the LiveCD of your choice.  But wouldn't it be nice if it were easy to install archlinux in this manner on several machines?  What if you could just download a compressed version of an up-to-date archlinux install and extract it on your hard drive?

With this in mind I have created an enhancement to the archbootstrap script called tarbuild.  Here is the usage:

archbootstrap <dir> [server]
archbootstrap -tb [--keep-build --use-build --with-cache] <dir> [server]

Without any of the long options, archbootstrap -tb will leave two items in the target directory: a base archlinux tarball around 76MB and a directory called pkg with the individual package tarballs used to build the install.  archbootstrap will use packages from this directory instead of downloading on subsequent runs if they are not updated in the current repository.  The tarball is datestamped for convenience.

--keep-build causes the uncompressed install to remain in the target directory in addition to the tarball.

--use-build creates a tarball of any existing install in the target directory.

--with-cache leaves the entire package cache in the tarball, more than doubling the size to about 160MB.

My overall intention for this project is for the Arch release engineering  team to periodically build base tarballs (weekly, for example) and push them out to the mirrors.  This way anyone can download and extract an up-to-date arch install without so much as burning a new CD.

The process is relatively quick if there is an up-to-date package cache in the pkg or var/cache/pacman/pkg directories in the target directory.  It's mostly I/O bound and takes 5 minutes on my laptop (4500rpm drive).

Here is the script:

#!/bin/bash
#########################################################################
# archbootstrap                                                         #
#                                                                       #
# archbootstrap is to archlinux what debootstrap is to Debian.          #
#                                                                       # 
#                                                                       #
# archbootstrap is public domain though i wouldn't mind getting credit. #
#                                                                       #
# by David Leinhäuser (archlinux@calavera.de)                           #
# Tarbuild mode and misc cleanups by butters (jsaretsk@andrew.cmu.edu)  #
# (minor tweak by dma174)                                               #
#########################################################################

########################
# Tarbuild Mode Switch #
########################
TARMODE=0
KBUILD=0
WCACHE=0
UBUILD=0

if [ "$1" == "-tb" ]; then
    INTERACTIVE=0
    TARMODE=1
    NUMPARAMS=0
    shift
    if [ "$1" == "--keep-build" ] || 
       [ "$2" == "--keep-build" ] ||
       [ "$3" == "--keep-build" ]; then
        KBUILD=1
        NUMPARAMS=$(( ${NUMPARAMS} + 1 ))
    fi
    if [ "$1" == "--with-cache" ] || 
       [ "$2" == "--with-cache" ] ||
       [ "$3" == "--with-cache" ]; then
        WCACHE=1
        NUMPARAMS=$(( ${NUMPARAMS} + 1 ))
    fi
    if [ "$1" == "--use-build" ] || 
       [ "$2" == "--use-build" ] ||
       [ "$3" == "--use-build" ]; then
        UBUILD=1
        NUMPARAMS=$(( ${NUMPARAMS} + 1 ))
    fi
    shift ${NUMPARAMS}
fi

####################
# Default Settings #
####################
MYVERSION="0.4"                              # Version number of this script.
GETCMD="${GETCMD:-wget --passive-ftp -c -q}" # What we use to download files.
PREINSTALL="${PREINSTALL:-glibc iputils}"    # Packages to install before using
                                             # pacman to fetch the rest.
IGNORE="${IGNORE:-}"                         # Packages not to be installed.
TARGET="${1:---help}"                        # Where to bootstrap to
SERVER="${2:-ftp://ftp.archlinux.org}"       # Where to fetch all the files.
INTERACTIVE="${INTERACTIVE:-1}"              # Interactive mode or batch mode

REPOS="current"                               # Which repository to use.
EXTRADIR="os/i686"
PKG_PACMAN="setup/pacman.pkg.tar.gz"         # Where to find pacman.
PKGLIST="setup/packages.txt"                 # Where to find the packages list.
PKG_PREFIX="base"                            # Install pkgs with this prefix.

####################
# Helper Functions #
####################
PREV_DIR="$(pwd)" # remember current directory so we can chdir back later.

# Print an error message to stderr and quit.
function error() {
    echo "Error: $*" >&2
    cd "${PREV_DIR}"
    exit 1;
}

# Simply exit the script.
function quit() {
    cd "${PREV_DIR}"
    exit 0;
}

# Wrapper around cd for lazy people like me.
function chngdir() {
    cd "$@" || error "chdir failed."
}

#Wrapper around mkdir for lazy people like me.
function makedir() {
    mkdir -p "$@" || error "mkdir failed."
}

#Fetch given URLs into current dir.
function fetch() {
    while [ -n "${1}" ]; do
        if [ -z "${1##file://*}" ]; then
            F="${1#file://}"
            echo "Copying ${F}."
            cp "${1#file://}" . || error "Error copying ${F}."
        else
            echo "Fetching ${1}."
            ${GETCMD} "${1}" || error "Error fetching ${1}."
        fi
        shift
    done
}

#Read the package list
PACKAGES=()
function read_pkglist() {
    while read -s; do
        PACKAGES=("${PACKAGES[@]}" "${REPLY}")
    done <"${BOOTSTRAP}/${PKGLIST##*/}"
}

#Return 1 if the first argument does not equal any of the other arguments.
function match() {
    PATTERN="${1}"
    shift
    while [ -n "${1}" ]; do
        [ "${PATTERN}" == "${1}" ] && return 0
        shift
    done
    return 1
}

######################
# Sanity checks etc. #
######################
case "${TARGET}" in
    --help|-h)
        echo "Usage: archbootstrap <dir> [server]"
        echo "Usage: archbootstrap -tb [--keep-build] [--use-build] [--with-cache] <dir> [server]"
        quit;
        ;;
    --version)
        echo "archbootstrap ${MYVERSION}";
        quit;
        ;;
esac

[ "$UID" == "0" ] || error "You need to be root."

echo "archbootstrap $MYVERSION"

# This is an easy way to get an useful absolute path.
chngdir "${TARGET}"
TARGET="$(pwd)"

# Create working directory.
TEMP="${TARGET}/tmp"
makedir -m 1777 -p "${TEMP}"
BOOTSTRAP="${TEMP}/bootstrap"
PKG_CACHE="${TARGET}/var/cache/pacman/pkg"
makedir -m 0755 -p "${BOOTSTRAP}"
makedir -m 0755 -p "${PKG_CACHE}"
if [ -d "${TARGET}/pkg" ]; then
    echo "Using existing pkg cache..."
    mv -f ${TARGET}/pkg/* "${PKG_CACHE}"
elif [ -n "$(ls ${PKG_CACHE})" ]; then
    echo "Using existing pkg cache..."
else
    echo "No pkg cache found, fetching all pkg files..."
fi

chngdir "${BOOTSTRAP}"

########################################
# Stuff we need for our staged install #
########################################
function set_stage() {
    STAGE="${1}"
}

STAGE="start"
if [ "${TARMODE}" == "1" ] && [ "${UBUILD}" == "1" ]; then
    STAGE="cleanup"
fi

##################
# pacman wrapper #
##################
PACMAN=("$(which chroot)" "${TARGET}" "${BOOTSTRAP#${TARGET}}/usr/bin/pacman.static")
function add_pkgs() {
    PKGS=()
    while [ -n "${1}" ]; do
        PKGS=("${PKGS[@]}" "${1#${TARGET}}")
        shift
    done
    "${PACMAN[@]}" -A "${PKGS[@]}" || error "Could not add a package."
}

function sync_pkgs() {
    if [ "${INTERACTIVE}" == "1" ]; then
        "${PACMAN[@]}" -Syf "${@}" || error "Could not sync a package."
    else
        echo y | "${PACMAN[@]}" -Syf "${@}" || 
            error "Could not sync a package."
    fi
}

#############
# Main loop #
#############
PREV_STAGE=""
FURL="${SERVER}/${REPOS}/${EXTRADIR}"

until [ "${PREV_STAGE}" == "${STAGE}" ]; do
    PREV_STAGE="${STAGE}"
    case "${STAGE}" in
    start)
        chngdir "${BOOTSTRAP}"
        fetch "${FURL}/${PKG_PACMAN}" "${FURL}/${PKGLIST}"
        tar xfz ${PKG_PACMAN##*/} || 
            error "Could not untar ${BOOTSTRAP}${PKG_PACMAN##*/}."
        makedir "${TARGET}/etc"
        { echo "[${REPOS}]";
          echo "Server = ${FURL}"; } >"${TARGET}/etc/pacman.conf" || 
            error "Could not create ${TARGET}/etc/pacman.conf."
        cp -f "/etc/resolv.conf" "${TARGET}/etc/resolv.conf" || 
            error "Could not copy /etc/resolv.conf."
        set_stage "pre_packages"
        ;;
    pre_packages)
        chngdir "${PKG_CACHE}"
        [ "${#PACKAGES[@]}" == "0" ] && read_pkglist;
        for I in "${PACKAGES[@]}"; do
            PKGPREFIX="${I%%/*}"
            PKGFILE="${I##*/}"
            PKGNAME="${PKGFILE%-*-*.pkg.tar.gz}"
            if match "${PKGNAME}" ${PREINSTALL}; then
                fetch "${FURL}/${PKGFILE}"
                add_pkgs "${PKG_CACHE}/${PKGFILE}"
            fi
        done
        set_stage "packages"
        ;;
    packages)
        chngdir "${BOOTSTRAP}"
        PKGS=()
        [ "${#PACKAGES[@]}" == "0" ] && read_pkglist;
        for I in "${PACKAGES[@]}"; do
            PKGPREFIX="${I%%/*}"
            PKGFILE="${I##*/}"
            PKGNAME="${PKGFILE%-*-*.pkg.tar.gz}"
            [ "${PKGPREFIX}" == "${PKG_PREFIX}" ] || continue
            match "${PKGNAME}" ${IGNORE} ${PREINSTALL} && continue
            PKGS=("${PKGS[@]}" "${PKGNAME}")
        done
        sync_pkgs "${PKGS[@]}"
        set_stage "cleanup"
        ;;
    cleanup)
        echo "Cleaning up..."
        chngdir "${TARGET}"
        rm -Rf "${BOOTSTRAP}" || error "Could not remove ${BOOTSTRAP}."
        [ "${TARMODE}" == "0" ] && quit
        set_stage "tarbuild"
        ;;
    tarbuild)
        echo "Building tarball..."
        chngdir "${TARGET}"
        if [ "${KBUILD}" == "0" ]; then
            TAROPTS="--remove-files --exclude ./pkg*"
            if [ "${WCACHE}" == "0" ]; then
                makedir "${TARGET}/pkg"
                mv -f ${PKG_CACHE}/* "${TARGET}/pkg"
            fi
        elif [ "${WCACHE}" == "0" ]; then
            TAROPTS="--exclude ./var/cache/pacman/pkg/* --exclude ./pkg*"
        else
            TAROPTS="--exclude ./pkg*"
        fi
        TARFILE="arch-base-`date +%Y%m%d`.tar.bz2"
        tar --create --bzip2 --preserve-permissions ${TAROPTS} --file="${TARGET}/${TARFILE}" ./* 
        if [ "${KBUILD}" == "0" ]; then
            for NODE in *; do
                [ -f "${NODE}" ] && continue
                [ "${NODE}" == "pkg" ] && continue
                rm -r "${NODE}"
            done
        fi
        quit
        ;;
    esac
done

Let me know if there is a better place to host this...

Offline

#2 2005-09-07 07:16:29

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: archbootstrap 0.4 with tarbuild

I've uploaded it here, in the place of the previous version - if anyone has problem copy-pasting, just grab it.

Thanks for your work, butters.

ps. Haven't tested it, but hope it works.
ps2. You could wikify it. There's some small reference to archbootstrap in the wiki, but I guess it could get separate article, possibly with your description which you've stated here.

Offline

#3 2005-09-07 08:57:14

butters
Member
Registered: 2005-08-31
Posts: 35

Re: archbootstrap 0.4 with tarbuild

It's fairly well tested, 4 out of the 7 possible combinations of options are confirmed working.  The other 3 are combinations I tested, with --use-build added.  This option just skips some some stages in the script, so it shouldn't make a difference.

I would warn against targeting a directory that has any non-build-related data anywhere inside of it, but the script will delete everything except first-level regular files and the pkg directory unless the --keep-build option is set. Even in this case, the resulting tarball will include whatever stuff you happened to have in there.  So, bottom line: don't target a directory that doesn't fit any of these descriptions:

1) empty directory
2) directory with a pkg subdir and/or existing tarballs
3) directory with a build from a previous archbootstrap
4) directory with both (2) and (3)

You will see two error-looking thingies in the output:

warning: /etc/resolv.conf saved as /etc/resolv.conf.pacorig
done.
sh: /usr/sbin/chroot: No such file or directory

and

warning: /etc/pacman.conf saved as /etc/pacman.conf.pacorig

I don't think they're problems, and they existed in the previous version of archbootstrap.  Please report any problems you experience in this thread.  Several of us know the script pretty well and can help you.  I don't expect any problems until the next time the repository layout changes... wink

Offline

#4 2005-09-07 09:07:58

dtw
Forum Fellow
From: UK
Registered: 2004-08-03
Posts: 4,439
Website

Re: archbootstrap 0.4 with tarbuild

butters, nice work, mate.  Surely you can test for 1-2 tho and query if they are not met?

Some thing like:

if [ ! -z $(find $target -type d ! -name /var) ] ; then
     echo "The target dir is not empty or contains unexpected files"
fi

Very simplistic but a minor check none the less.  If you have an exact list of dirs and files (I assume archbootstrap-*.tar.gz and /var/cache/pacman/pkg) and you find anything other than those display the warning.

Files is easiest...

find * -maxdepth 0 -type f ! -name "archbootstrap-*.tar.gz"

Yeah, random code snippets, which I'm sure you can implement yourself but just to make the point that a sanity check for anything other than the correct files would be pretty easy... not too restrictive to have a warning i think smile

Offline

#5 2005-09-08 06:22:28

butters
Member
Registered: 2005-08-31
Posts: 35

Re: archbootstrap 0.4 with tarbuild

If you have an exact list of dirs and files (I assume archbootstrap-*.tar.gz and /var/cache/pacman/pkg) and you find anything other than those display the warning.

Well, like I said it won't ever remove top-level regular files or the pkg directory.  I could make an exhaustive list of top-level directories to remove in case --keep-build is not set (/bin, /boot, /etc, /dev...), but this would be a nasty hack and wouldn't accomodate any change in top-level directory structure.  Also, for obvious reasons no sanity check is effective if the user specifies / as their target.  That would be a cool experiment, though.

It's pretty simple: create a *leaf* directory and use that as the target unless you really know what you're doing.  If you specify /usr as your target directory, then bad things will happen.

The primary audience for this script is advanced users (to install from the host environment of their choice), system administrators (to create an install image for multiple machines), and Arch release engineering (to generate and mirror up-to-date base tarballs).  This audience is expected to realize that one should choose/create a leaf directory whenever they build software.

If any of this confuses you, and you still want to use archbootstrap, I would suggest using /var/tmp as your target directory.  You don't even need to be root to build in this directory.

Offline

#6 2005-09-08 18:20:40

Riklaunim
Member
Registered: 2005-04-09
Posts: 106
Website

Re: archbootstrap 0.4 with tarbuild

I'm installing Arch from CentOS right now... can't stand its quality... RedHat sucks smile
1. I can't type "e" in any term smile
2. cut down bash (no chroot, other - archbootstrap fails...)
3. small ammount of packages... no firebird DB etc.
Currently Arch is installing via chrooted gentoo stage3 in which archbootstrap works... now im chrooted in arch from chrooted gentoo 8) )

Offline

#7 2005-09-15 01:30:09

deficite
Member
From: Augusta, GA
Registered: 2005-06-02
Posts: 693

Re: archbootstrap 0.4 with tarbuild

I would say this is my favorite thing to come about in Arch Linux since the AUR. Thanks a lot for this great script!

Offline

#8 2005-09-16 01:17:58

butters
Member
Registered: 2005-08-31
Posts: 35

Re: archbootstrap 0.4 with tarbuild

Hey, I could use someone's help in understanding a weird scripting anomaly:

Well, I finally got around to using my script for an actual install, and it's not working... sort of.  Everything's actually *working*, but it triggers the error function (killing execution) even when the command succeeds.  I'm only experiencing this when running from the Gentoo LiveCD.  Here's an example:

tar xfz ${PKG_PACMAN##*/} ||
error "Could not untar ${BOOTSTRAP}${PKG_PACMAN##*/}."

The tar command succeeds and returns 0, I can verify this by echoing the $? variable.  The tarball untars as it's supposed to, but it triggers the error function anyway.  The funny thing is, this actually makes sense:

0 || 0 = 0
0 || 1 = 1

Therefore the shell interpreter *should* be executing the second half of the compound conditional in the case where the first expression evaluates to 0.  I can't understand why the above snippet (and various other instances in the archbootstrap script) actually works on many shells!!

To me, I should be replacing these || checks with && checks.  Am I going insane?  Can someone shed some light on this issue?

Offline

#9 2005-09-16 01:59:00

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: archbootstrap 0.4 with tarbuild

what you have should work with "||".
I noticed you have an escape character "" at the end of your tar statement. If you add additional spaces after it, the value of the escape character (as well as the logical or) is lost and will execute your error function.

In these kinds of bash statements, think of logical or as "if the first command does not exit true, do the next."
Logical and is "if the first command exits true, do the next."

Offline

#10 2005-09-16 02:54:31

butters
Member
Registered: 2005-08-31
Posts: 35

Re: archbootstrap 0.4 with tarbuild

I'll check for the trailing space, but that wouldn't explain why archbootstrap works on one machine and not on another.  Also, in the case of most CLI tools, a successful exit is indicated by a false (0) return code.  Therefore the logical OR (to me) means, if the command exits successfully, then execute what's next.  If the command exits unsuccessfully (returning nonzero), then the outcome of the compound conditional is determinably 'true', so don't execute the second expression.

Interestingly, GCC automatically strips trailing whitespace in the case where a backslash is the last nonwhitespace character on a line.  I guess bash does not.  Maybe bash 3.0.x does this and 2.95.x does not... I think I might be on to something.

Offline

#11 2005-09-16 04:32:43

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: archbootstrap 0.4 with tarbuild

I guess I'm not understanding you correctly. What you're saying is not making anysense to me, but you've obviously done things right with logical ors and ands after a quick gander at your code.

You say this is only happening in the Gentoo LiveCD? Maybe its linking to a different shell?
Maybe you could put this into an if statement instead?

if [ $? -ne 0 ]; then
error ""
fi

Offline

#12 2005-09-17 03:20:41

deficite
Member
From: Augusta, GA
Registered: 2005-06-02
Posts: 693

Re: archbootstrap 0.4 with tarbuild

Did you download the script from lucke's link? I tried using that and it gave me the same thing. If so, try copy and pasting the script from what butters provided.

Offline

#13 2005-09-17 03:35:56

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: archbootstrap 0.4 with tarbuild

deficite wrote:

Did you download the script from lucke's link? I tried using that and it gave me the same thing. If so, try copy and pasting the script from what butters provided.

Yes, looking at the link from lucke I see there is spaces after the escape characters as I mentioned earlier. That would explain the problems. Remove them and it should work.

Offline

#14 2005-09-17 07:04:37

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: archbootstrap 0.4 with tarbuild

I've removed the spaces from the hosted file (I hope, at least).

Offline

#15 2005-09-23 06:29:31

changcheh
Member
Registered: 2005-09-23
Posts: 2

Re: archbootstrap 0.4 with tarbuild

I am trying to do this from within Vector Linux to /dev/hda1 which I have mounted as /mnt/deb.  I have removed all the files from this folder with "rm -r *".  I try to run the script which I downloaded from luckies link with the command

./archbootstrap /mnt/deb/ http://darkstar.ist.utl.pt/archlinux

It gets so far and then prints this error message

archbootstrap 0.4
No pkg cache found, fetching all pkg files...
Fetching http://darkstar.ist.utl.pt/archlinux/cu … pkg.tar.gz.
Fetching http://darkstar.ist.utl.pt/archlinux/cu … ckages.txt.
Fetching http://darkstar.ist.utl.pt/archlinux/cu … pkg.tar.gz.
warning: cannot find chroot binary - unable to run scriptlets
loading package data... done.
checking for file conflicts... done.
installing glibc...
could not extract usr/sbin/zic: Invalid argument
errors occurred while installing glibc
done.

any ideas?

If not I have a downloaded .iso, is it possible to somehow use the packages from this instead of a remote ftp site?

thanks
jonathan


a game of chess is like a swordfight, you must think first before you move

Offline

#16 2005-09-23 10:27:22

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: archbootstrap 0.4 with tarbuild

Erm, maybe you don't have chroot?

'which chroot' to check.

Offline

#17 2005-09-23 13:49:50

changcheh
Member
Registered: 2005-09-23
Posts: 2

Re: archbootstrap 0.4 with tarbuild

Thanks, but I have chroot in /usr/bin/chroot
I checked the script to see if I could change the path to it, but it only calls 'which chroot'


a game of chess is like a swordfight, you must think first before you move

Offline

#18 2005-09-30 16:20:48

dlin
Member
From: Taipei,Taiwan
Registered: 2005-09-21
Posts: 265

Re: archbootstrap 0.4 with tarbuild

butters wrote:

This is a continuation of a previous thread:

http://bbs.archlinux.org/viewtopic.php?t=4020

In short, archbootstrap lets you install a base archlinux system to the target directory of your choice from within any remotely standard *nix system, including the LiveCD of your choice.  But wouldn't it be nice if it were easy to install archlinux in this manner on several machines?  What if you could just download a compressed version of an up-to-date archlinux install and extract it on your hard drive?

Hi, could you put this util into AUR?
And also added a line on http://wiki.archlinux.org/index.php/Use … utionsPage


Running 4 ArchLinux including sh4twbox,server,notebook,desktop. my AUR packages

Offline

#19 2005-11-13 11:20:00

dtw
Forum Fellow
From: UK
Registered: 2004-08-03
Posts: 4,439
Website

Re: archbootstrap 0.4 with tarbuild

If someone could check my bash I'd like to add a patch to this:

--- archbootstrap    2005-11-05 09:08:30.000000000 +0000
+++ archbootstrap.new    2005-11-13 11:16:31.000000000 +0000
@@ -153,7 +153,8 @@
  TEMP="${TARGET}/tmp" 
  makedir -m 1777 -p "${TEMP}" 
  BOOTSTRAP="${TEMP}/bootstrap" 
- PKG_CACHE="${TARGET}/var/cache/pacman/pkg" 
+ PKG_CACHE="${TARGET}/var/cache/pacman/pkg"
+ LOCAL_CACHE="/var/cache/pacman/pkg"
  makedir -m 0755 -p "${BOOTSTRAP}" 
  makedir -m 0755 -p "${PKG_CACHE}" 
  if [ -d "${TARGET}/pkg" ]; then 
@@ -161,8 +162,12 @@
     mv -f ${TARGET}/pkg/* "${PKG_CACHE}" 
  elif [ -n "$(ls ${PKG_CACHE})" ]; then 
     echo "Using existing pkg cache..." 
+ elif [ -d "${LOCAL_CACHE}" ]; then
+    echo "Local pkg cache found, using where possible..."
+    USE_LOCAL="true"
  else 
-    echo "No pkg cache found, fetching all pkg files..." 
+    echo "No pkg cache found, fetching all pkg files..."
+    USE_LOCAL=
  fi 
  
  chngdir "${BOOTSTRAP}" 
@@ -230,9 +235,14 @@
           PKGPREFIX="${I%%/*}" 
           PKGFILE="${I##*/}" 
           PKGNAME="${PKGFILE%-*-*.pkg.tar.gz}" 
-          if match "${PKGNAME}" ${PREINSTALL}; then 
-             fetch "${FURL}/${PKGFILE}" 
-             add_pkgs "${PKG_CACHE}/${PKGFILE}" 
+          if match "${PKGNAME}" ${PREINSTALL}; then
+          if [ ! -z $LOCAL_CACHE ]; then
+              echo "Copying ${PKGFILE} from $LOCAL_CACHE"
+              [ -e "${LOCAL_CACHE}/${PKGFILE}" ] && cp "${LOCAL_CACHE}/${PKGFILE}" "${PKG_CACHE}/"
+          else
+              fetch "${FURL}/${PKGFILE}"
+          fi
+          add_pkgs "${PKG_CACHE}/${PKGFILE}"
           fi 
        done 
        set_stage "packages" 
@@ -246,9 +256,13 @@
           PKGFILE="${I##*/}" 
           PKGNAME="${PKGFILE%-*-*.pkg.tar.gz}" 
           [ "${PKGPREFIX}" == "${PKG_PREFIX}" ] || continue 
-          match "${PKGNAME}" ${IGNORE} ${PREINSTALL} && continue 
+          match "${PKGNAME}" ${IGNORE} ${PREINSTALL} && continue
+      if [ ! -z $LOCAL_CACHE ]; then
+           echo "Copying ${PKGFILE} from $LOCAL_CACHE"
+           [ -e "${LOCAL_CACHE}/${PKGFILE}" ] && cp "${LOCAL_CACHE}/${PKGFILE}" "${PKG_CACHE}/"
+          fi
           PKGS=("${PKGS[@]}" "${PKGNAME}") 
-       done 
+       done
        sync_pkgs "${PKGS[@]}" 
        set_stage "cleanup" 
        ;; 

It basically uses pkgs from your local cache if it exists and they exist in it - only useful for bootstraping a new arch install from within an arch install but useful none the less!

Offline

#20 2006-06-01 19:46:43

quetzal
Member
From: Germany
Registered: 2004-05-05
Posts: 19

Re: archbootstrap 0.4 with tarbuild

butters wrote:

Hey, I could use someone's help in understanding a weird scripting anomaly:

Well, I finally got around to using my script for an actual install, and it's not working... sort of.  Everything's actually *working*, but it triggers the error function (killing execution) even when the command succeeds.  I'm only experiencing this when running from the Gentoo LiveCD.  Here's an example:

tar xfz ${PKG_PACMAN##*/} ||
error "Could not untar ${BOOTSTRAP}${PKG_PACMAN##*/}."

The tar command succeeds and returns 0, I can verify this by echoing the $? variable.  The tarball untars as it's supposed to, but it triggers the error function anyway.  The funny thing is, this actually makes sense:

0 || 0 = 0
0 || 1 = 1

Therefore the shell interpreter *should* be executing the second half of the compound conditional in the case where the first expression evaluates to 0.  I can't understand why the above snippet (and various other instances in the archbootstrap script) actually works on many shells!!

To me, I should be replacing these || checks with && checks.  Am I going insane?  Can someone shed some light on this issue?

Hello, it's a realy old Thread - but maybe somebody have the same problem.

My solution for this issue:  i cut off all spaces and all "" now it's work.
But i don't understand - because - you have use the "" befor (at sync) - at this place it works

Here the "new" wink Version:

#!/bin/bash 
######################################################################### 
# archbootstrap                                                         # 
#                                                                       # 
# archbootstrap is to archlinux what debootstrap is to Debian.          # 
#                                                                       # 
#                                                                       # 
# archbootstrap is public domain though i wouldn't mind getting credit. # 
#                                                                       # 
# by David Leinhäuser (archlinux@calavera.de)                           # 
# Tarbuild mode and misc cleanups by butters (jsaretsk@andrew.cmu.edu)  # 
# (minor tweak by dma174)                                               # 
######################################################################### 

######################## 
# Tarbuild Mode Switch # 
######################## 
#set -xv

TARMODE=0 
KBUILD=0 
WCACHE=0 
UBUILD=0 

if [ "$1" == "-tb" ]; then 
   INTERACTIVE=0 
   TARMODE=1 
   NUMPARAMS=0 
   shift 
   if [ "$1" == "--keep-build" ] || 
      [ "$2" == "--keep-build" ] || 
      [ "$3" == "--keep-build" ]; then 
      KBUILD=1 
      NUMPARAMS=$(( ${NUMPARAMS} + 1 )) 
   fi 
   if [ "$1" == "--with-cache" ] || 
      [ "$2" == "--with-cache" ] || 
      [ "$3" == "--with-cache" ]; then 
      WCACHE=1 
      NUMPARAMS=$(( ${NUMPARAMS} + 1 )) 
   fi 
   if [ "$1" == "--use-build" ] || 
      [ "$2" == "--use-build" ] || 
      [ "$3" == "--use-build" ]; then 
      UBUILD=1 
      NUMPARAMS=$(( ${NUMPARAMS} + 1 )) 
   fi 
   shift ${NUMPARAMS} 
fi 

#################### 
# Default Settings # 
#################### 
MYVERSION="0.4"                              # Version number of this script. 
GETCMD="${GETCMD:-wget --passive-ftp -c -q}" # What we use to download files. 
PREINSTALL="${PREINSTALL:-glibc iputils}"    # Packages to install before using 
                                             # pacman to fetch the rest. 
IGNORE="${IGNORE:-}"                         # Packages not to be installed. 
TARGET="${1:---help}"                        # Where to bootstrap to 
SERVER="${2:-ftp://ftp.archlinux.org}"       # Where to fetch all the files. 
INTERACTIVE="${INTERACTIVE:-1}"              # Interactive mode or batch mode 

REPOS="current"                              # Which repository to use. 
EXTRADIR="os/i686" 
PKG_PACMAN="setup/pacman.pkg.tar.gz"         # Where to find pacman. 
PKGLIST="setup/packages.txt"                 # Where to find the packages list. 
PKG_PREFIX="base"                            # Install pkgs with this prefix. 

#################### 
# Helper Functions # 
#################### 
PREV_DIR="$(pwd)" # remember current directory so we can chdir back later. 

# Print an error message to stderr and quit. 
function error() { 
   echo "Error: $*" >&2 
   cd "${PREV_DIR}" 
   exit 1; 
} 

# Simply exit the script. 
function quit() { 
   cd "${PREV_DIR}" 
   exit 0; 
} 

# Wrapper around cd for lazy people like me. 
function chngdir() { 
   cd "$@" || error "chdir failed." 
} 

#Wrapper around mkdir for lazy people like me. 
function makedir() { 
   mkdir -p "$@" || error "mkdir failed." 
} 

#Fetch given URLs into current dir. 
function fetch() { 
   while [ -n "${1}" ]; do 
      if [ -z "${1##file://*}" ]; then 
         F="${1#file://}" 
         echo "Copying ${F}." 
         cp "${1#file://}" . || error "Error copying ${F}." 
      else 
         echo "Fetching ${1}." 
         ${GETCMD} "${1}" || error "Error fetching ${1}." 
      fi 
      shift 
   done 
} 

#Read the package list 
PACKAGES=() 
function read_pkglist() { 
   while read -s; do 
      PACKAGES=("${PACKAGES[@]}" "${REPLY}") 
   done <"${BOOTSTRAP}/${PKGLIST##*/}" 
} 

#Return 1 if the first argument does not equal any of the other arguments. 
function match() { 
   PATTERN="${1}" 
   shift 
   while [ -n "${1}" ]; do 
      [ "${PATTERN}" == "${1}" ] && return 0 
      shift 
   done 
   return 1 
} 

###################### 
# Sanity checks etc. # 
###################### 
case "${TARGET}" in 
   --help|-h) 
      echo "Usage: archbootstrap <dir> [server]" 
      echo "Usage: archbootstrap -tb [--keep-build] [--use-build] [--with-cache] <dir> [server]" 
      quit; 
      ;; 
   --version) 
      echo "archbootstrap ${MYVERSION}"; 
      quit; 
      ;; 
esac 

[ "$UID" == "0" ] || error "You need to be root." 

echo "archbootstrap $MYVERSION" 

# This is an easy way to get an useful absolute path. 
chngdir "${TARGET}" 
TARGET="$(pwd)" 

# Create working directory. 
TEMP="${TARGET}/tmp" 
makedir -m 1777 -p "${TEMP}" 
BOOTSTRAP="${TEMP}/bootstrap" 
PKG_CACHE="${TARGET}/var/cache/pacman/pkg" 
makedir -m 0755 -p "${BOOTSTRAP}" 
makedir -m 0755 -p "${PKG_CACHE}" 
if [ -d "${TARGET}/pkg" ]; then 
   echo "Using existing pkg cache..." 
   mv -f ${TARGET}/pkg/* "${PKG_CACHE}" 
elif [ -n "$(ls ${PKG_CACHE})" ]; then 
   echo "Using existing pkg cache..." 
else 
   echo "No pkg cache found, fetching all pkg files..." 
fi 

chngdir "${BOOTSTRAP}" 

######################################## 
# Stuff we need for our staged install # 
######################################## 
function set_stage() { 
   STAGE="${1}" 
} 

STAGE="start" 
if [ "${TARMODE}" == "1" ] && [ "${UBUILD}" == "1" ]; then 
   STAGE="cleanup" 
fi 

################## 
# pacman wrapper # 
################## 
PACMAN=("$(which chroot)" "${TARGET}" "${BOOTSTRAP#${TARGET}}/usr/bin/pacman.static") 
function add_pkgs() { 
   PKGS=() 
   while [ -n "${1}" ]; do 
      PKGS=("${PKGS[@]}" "${1#${TARGET}}") 
      shift 
   done 
   "${PACMAN[@]}" -A "${PKGS[@]}" || error "Could not add a package." 
} 

function sync_pkgs() { 
   if [ "${INTERACTIVE}" == "1" ]; then 
      "${PACMAN[@]}" -Syf "${@}" || error "Could not sync a package." 
   else 
      echo y | "${PACMAN[@]}" -Syf "${@}" ||  
         error "Could not sync a package." 
   fi 
} 

############# 
# Main loop # 
############# 
PREV_STAGE="" 
FURL="${SERVER}/${REPOS}/${EXTRADIR}" 

until [ "${PREV_STAGE}" == "${STAGE}" ]; do 
   PREV_STAGE="${STAGE}" 
   case "${STAGE}" in 
   start) 
      chngdir "${BOOTSTRAP}" 
      fetch "${FURL}/${PKG_PACMAN}" "${FURL}/${PKGLIST}" 
      tar xzf ${PKG_PACMAN##*/}|| error "Could not untar ${BOOTSTRAP}${PKG_PACMAN##*/}." 
      makedir "${TARGET}/etc" 
      { echo "[${REPOS}]"; 
        echo "Server = ${FURL}"; } >"${TARGET}/etc/pacman.conf"|| error "Could not create ${TARGET}/etc/pacman.conf." 
      cp -f "/etc/resolv.conf" "${TARGET}/etc/resolv.conf"|| error "Could not copy /etc/resolv.conf." 
      set_stage "pre_packages" 
      ;; 
   pre_packages) 
      chngdir "${PKG_CACHE}" 
      [ "${#PACKAGES[@]}" == "0" ] && read_pkglist; 
      for I in "${PACKAGES[@]}"; do 
         PKGPREFIX="${I%%/*}" 
         PKGFILE="${I##*/}" 
         PKGNAME="${PKGFILE%-*-*.pkg.tar.gz}" 
         if match "${PKGNAME}" ${PREINSTALL}; then 
            fetch "${FURL}/${PKGFILE}" 
            add_pkgs "${PKG_CACHE}/${PKGFILE}" 
         fi 
      done 
      set_stage "packages" 
      ;; 
   packages) 
      chngdir "${BOOTSTRAP}" 
      PKGS=() 
      [ "${#PACKAGES[@]}" == "0" ] && read_pkglist; 
      for I in "${PACKAGES[@]}"; do 
         PKGPREFIX="${I%%/*}" 
         PKGFILE="${I##*/}" 
         PKGNAME="${PKGFILE%-*-*.pkg.tar.gz}" 
         [ "${PKGPREFIX}" == "${PKG_PREFIX}" ]|| continue 
         match "${PKGNAME}" ${IGNORE} ${PREINSTALL}&& continue 
         PKGS=("${PKGS[@]}" "${PKGNAME}") 
      done 
      sync_pkgs "${PKGS[@]}" 
      set_stage "cleanup" 
      ;; 
   cleanup) 
      echo "Cleaning up..." 
      chngdir "${TARGET}" 
      rm -Rf "${BOOTSTRAP}"|| error "Could not remove ${BOOTSTRAP}." 
      [ "${TARMODE}" == "0" ] && quit 
      set_stage "tarbuild" 
      ;; 
   tarbuild) 
      echo "Building tarball..." 
      chngdir "${TARGET}" 
      if [ "${KBUILD}" == "0" ]; then 
         TAROPTS="--remove-files --exclude ./pkg*" 
         if [ "${WCACHE}" == "0" ]; then 
            makedir "${TARGET}/pkg" 
            mv -f ${PKG_CACHE}/* "${TARGET}/pkg" 
         fi 
      elif [ "${WCACHE}" == "0" ]; then 
         TAROPTS="--exclude ./var/cache/pacman/pkg/* --exclude ./pkg*" 
      else 
         TAROPTS="--exclude ./pkg*" 
      fi 
      TARFILE="arch-base-`date +%Y%m%d`.tar.bz2" 
      tar --create --bzip2 --preserve-permissions ${TAROPTS} --file="${TARGET}/${TARFILE}" ./* 
      if [ "${KBUILD}" == "0" ]; then 
         for NODE in *; do 
            [ -f "${NODE}" ] && continue 
            [ "${NODE}" == "pkg" ] && continue 
            rm -r "${NODE}" 
         done 
      fi 
      quit 
      ;; 
   esac 

hope this helps
btw: the link from linky doesn't work

Offline

#21 2006-06-01 20:07:19

quetzal
Member
From: Germany
Registered: 2004-05-05
Posts: 19

Re: archbootstrap 0.4 with tarbuild

But now i got some other errors with "chroot":

Fetching ftp://ftp.archlinux.org/current/os/i686/iputils-021109-4.pkg.tar.gz.
warning: cannot find chroot binary - unable to run scriptlets

But i have chroot!

Edit: ok, it seems to be all installed anyhow - but what is "scriptlets" - i don't know from where i got this message.

Offline

#22 2006-06-02 10:03:16

quetzal
Member
From: Germany
Registered: 2004-05-05
Posts: 19

Re: archbootstrap 0.4 with tarbuild

fuck - now i get the message:

 Warning: unable to open an initial console

and can't boot

I use a Kernel 2.6.16-18 downloaded from: http://uml.nagafix.co.uk/

I think it's better to go to another Distribution for UML. (maybe debian)

Offline

#23 2006-06-21 13:59:14

tom5760
Member
From: Philadelphia, PA, USA
Registered: 2006-02-05
Posts: 283
Website

Re: archbootstrap 0.4 with tarbuild

I am having the same problem.  I used the bootstrap script to install Arch from a knoppix livecd and I got that

warning: cannot find chroot binary - unable to run scriptlets 

line.  Once I isntalled my kernel, configured everything, and rebooted I got this message:

 Warning: unable to open an initial console

Anyone have any ideas on how to fix it?

Offline

#24 2006-06-21 14:35:55

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: archbootstrap 0.4 with tarbuild

Try running 'migrate-udev'.

Offline

#25 2006-06-21 15:05:08

tom5760
Member
From: Philadelphia, PA, USA
Registered: 2006-02-05
Posts: 283
Website

Re: archbootstrap 0.4 with tarbuild

It worked!  Thanks a lot!

Offline

Board footer

Powered by FluxBB