Pacman Optimization - Part 2

ody · 2006-04-06 17:05:37

So people notice I now have a decent script for this here it is:

#!/bin/bash
#
#   pacman-cage
#
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8'
dbroot="/var/lib/pacman"
pacmandb="/var/lib/pacman.db"

usage() {
        echo "pacman-cage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-cage creates a loopbacked filesystem in a contigious file."
        echo "This will give better response times when using pacman"
        echo
}

die() {
        echo "pacman-cage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

if [ "`id -u`" != 0 ]; then
        die "You must be root to cage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi
# make sure pacman.db hasnt already been made
if [ -f $pacmandb ]; then
        die "$pacmandb already exists!."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory"
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck
 
# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> creating pacman.db loopback file..."
dd if=/dev/zero of=$pacmandb bs=1M count=150 > /dev/null 2>&1

echo "==> creating ext2 -O dir_index -b 1024 -m 0 on $pacmandb..."
yes | mkfs.ext2 -O dir_index -b 1024 -m 0 $pacmandb > /dev/null 2>&1
       
echo "==> creating temporary mount point /mnt/tmp-pacman.."
mkdir /mnt/tmp-pacman

echo "==> mounting pacman.db to temporary mount point..."
mount -o loop $pacmandb /mnt/tmp-pacman

echo "==> copying pacman database to temporary mount point..."
cp -a /var/lib/pacman/. /mnt/tmp-pacman

echo "==> unmounting temporary mount point..."
umount /mnt/tmp-pacman

echo "==> removing temporary mount point..."
rmdir /mnt/tmp-pacman
       
echo "==> moving old /var/lib/pacman to /var/lib/pacman.bak..."
mv /var/lib/pacman /var/lib/pacman.bak
       
echo "==> createing new pacman db mount point @ $dbroot..."
mkdir $dbroot

echo "==> Mounting new pacman db..."
mount -o loop $pacmandb $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        umount $dbroot
        rm $pacmandb
        mv $dbroot.bak $dbroot
        die_r "integrity check FAILED, reverting to old database"
fi

echo "==> Updating /etc/fstab to reflect changes..."
echo "$pacmandb $dbroot ext2 loop,defaults 0 0" >> /etc/fstab
 
rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new
 
echo
echo "Finished.  Your pacman database has been caged!.  May the speedy pacman be with you."
echo

exit 0

lilsirecho · 2006-04-06 17:26:52

ody;

To try this new program in view of the present older version already in place, what steps are advised?

I assume this program is run directly from root prompt.

ody · 2006-04-06 17:33:40

lilsirecho wrote:

ody;
To try this new program in view of the present older version already in place, what steps are advised?
I assume this program is run directly from root prompt.

I'm just sorting out a pacman-uncage as we speak. brb.

ody · 2006-04-07 16:42:49

OK, heres the uncage script:

#!/bin/bash
#
#   pacman-uncage
#
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8'
dbroot="/var/lib/pacman"
tmproot="/var/lib/pacman.new"
pacmandb="/var/lib/pacman.db"

usage() {
        echo "pacman-uncage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-uncage returns your pacman db to the generic style."
        echo
}

die() {
        echo "pacman-uncage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

if [ "`id -u`" != 0 ]; then
        die "You must be root to uncage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory"
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck

# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> copying pacman.db contents back, note: the time needed to get a brew is now."
mkdir $tmproot
cp -a $dbroot/. $tmproot

echo "==> unmounting old dbroot and moving new one in"
umount $dbroot
rmdir $dbroot
mv $tmproot $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        rm -rf $dbroot
        mkdir $dbroot
        mount -a
        die_r "integrity check FAILED, reverting to old database"
fi

echo "==> Removing old pacman.db"
rm $pacmandb

rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new
            
echo
echo "Finished.  Your pacman database has been uncaged!. Welcome home."
echo "You will need to remove the old mount line from your /etc/fstab"
echo

exit 0

lilsirecho · 2006-04-07 17:41:48

Am I to assume that this new uncage program applies to the original posting of pacman-cage and the newer one the subject of this thread.

To utilize this uncage program, I would appreciate the correct entries to be made to start this new version.

Your programs have been quite well accepted by arch users. Very nice!!!

wain · 2006-04-07 17:52:53

Very good !

Do not forget to build kernel with Loopback device support (BLK_DEV_LOOP)

ody · 2006-04-07 18:29:40

lilsirecho wrote:

Am I to assume that this new uncage program applies to the original posting of pacman-cage and the newer one the subject of this thread.
To utilize this uncage program, I would appreciate the correct entries to be made to start this new version.
Your programs have been quite well accepted by arch users. Very nice!!!

The uncage will work with the first version of pacman-cage back in the first thread, first post.

wain · 2006-04-07 19:05:48

Just a little test with my old 350Mhz Celeron Notebook 64mb ram and ext3+dir_index:

before pacman-cage wrote:

# pacman-optimize
# time pacman -Ss pacman
...
real 0m58.469s
user 0m2.279s
sys 0m3.961s

after pacman-cage wrote:

# time pacman -Ss pacman
...
real 0m22.234s
user 0m2.250s
sys 0m3.181s

Amazing 8)
Thank you ody !

edit: it's an old notebook

ody · 2006-04-07 19:21:38

wain wrote:

Just a little test with my old 350Mhz Celeron Laptop 64mb in ext3+dir_index:
before wrote:
# pacman-optimize
# time pacman -Ss pacman
...
real 0m58.469s
user 0m2.279s
sys 0m3.961s
after wrote:
# time pacman -Ss pacman
...
real 0m22.234s
user 0m2.250s
sys 0m3.181s
Amazing 8)
Thank you ody !

I think your useing Judds pacman-optimize, I didn't make that, its part of the pacman package! my process is different, try the script in the first post and see how it fairs.

wain · 2006-04-07 19:41:05

no, I just used pacman-optimize before making the first test (58sec).

The test "after" (22sec) is made with pacman-cage ! 8)

pacman-cage is even better than what you think ! :shock:

ody · 2006-04-07 19:45:35

wain wrote:

no, I just used pacman-optimize before making the first test (58sec).
The test "after" (22sec) is made with pacman-cage ! 8)

Ah right, sorry! glad its helped your celeron out! cheers for the stats aswell, nice to know it makes a real difference.

Gullible Jones · 2006-04-07 20:15:52

Hey Wain, are you using XFS? Because that's the only filesystem I know of where pacman -Ss takes more than 15 seconds on a decent system.

lilsirecho · 2006-04-07 20:33:16

ody;
Ran time pgm on original pacman-cage:

=================================================

[root@n6re ~]# time pacman -Ss pacman
current/pacman 2.9.8-1
A .tar.gz based package manager with dependency support
extra/namcap 1.5.3.1-1
A Pacman package analyzer
extra/srcpac 0.4.1-1
The pacman from-source wrapper
community/gtkpacman 1.2-2
A frontend written in python, using pygtk, to pacman, the archlinux package
manager
community/jacman 0.3.1-2
Java-based GUI front-end for pacman

real 0m0.782s
user 0m0.420s
sys 0m0.320s
[root@n6re ~]#

================================================

?????? :?: :?:

wain · 2006-04-07 21:11:45

Gullible Jones wrote:

Hey Wain, are you using XFS? Because that's the only filesystem I know of where pacman -Ss takes more than 15 seconds on a decent system.

Is a celeron 350Mhz with 64mb ram a decent system for you ?
I'm using ext3 with dir_index. Best choice for this old notebook

lucke · 2006-04-07 21:50:58

Weee.
I fscked up, somehow.

Does anyone know how to install packages from a given directory using pacman -S (so it'd resolve dependencies)? Command involves using ls, sed/cut/whatever and xargs, but I can't find the final solution.

lucke · 2006-04-07 21:53:03

Weee.
I fscked up, somehow.

Does anyone know how to install packages from a given directory using pacman -S (so it'd resolve dependencies)? Command involves using ls, sed/cut/whatever and xargs, but I can't find the final solution.

ody · 2006-04-07 22:10:27

xhectorx wrote:

ok the solution was to move /var/lib/pacman.bak to /var/lib/pacman i tought the script had done this.
(pacman-cage: integrity check FAILED, reverting to old database)
well anyways..

I havnt finalised the script really.. and never thought I'd see the day Linux was built without ext2 support.. anyway atleast it created a backup, you will also have a redundant db @ /mnt/tmp-pacman that you need to remove.

I will put better sanity checking in place.

Sander · 2006-04-08 11:09:41

I get the following when I run the script:

sander ~ > sudo exec/pacman-cage
==> md5sum'ing the old database...
==> creating pacman.db loopback file...
==> creating ext2 -O dir_index -b 1024 -m 0 on /var/lib/pacman.db...
==> creating temporary mount point /mnt/tmp-pacman..
==> mounting pacman.db to temporary mount point...
ioctl: LOOP_CLR_FD: Apparaat of bron bezig ((Device or resource busy))
mount: u moet de bestandssysteem soort aangeven ((you have to indicate the fs type))
==> copying pacman database to temporary mount point...
==> unmounting temporary mount point...
umount: /mnt/tmp-pacman: niet aangekoppeld ((not mounted))
==> removing temporary mount point...
rmdir: `/mnt/tmp-pacman': Map niet leeg ((directory not empty))
==> moving old /var/lib/pacman to /var/lib/pacman.bak...
==> createing new pacman db mount point @ /var/lib/pacman...
==> Mounting new pacman db...
ioctl: LOOP_CLR_FD: Apparaat of bron bezig ((Device or resource busy))
mount: u moet de bestandssysteem soort aangeven ((you have to indicate the fs type))
==> md5sum'ing the new database...
==> checking integrity...
umount: /var/lib/pacman: niet aangekoppeld ((not mounted))
pacman-cage: integrity check FAILED, reverting to old database

I added translations from Dutch in (())'s where necessary.

I googled for the error but all I found was about people trying to mount their initrd images as loopback devices or something =/ anyways. I'm using the 2.6.15-archck kernel. Any ideas, anyone? I'd really like to try this, as I have to run pacman-optimize almost every 2 days to get decent searching speed

additional note: if I try to mount an iso file (I randomly chose the arch 0.7.1 iso here) it works just fine =/

wain · 2006-04-08 22:10:44

Why not doing the same job for /var/abs/ directory ?

lilsirecho · 2006-04-09 15:37:39

ody;

On page one of this thread, there are several listings of results obtaines using ...pacman -Ss pacman.

I entered my results as well and am wondering if the test is valid due to the very stark difference in the time reported for the three elements, my values are less than a second, in case you didn't notice.

EDIT: I unmounted the /var/lib/pacman.db and proved that it is actively running pacman when mounted.

cmf · 2006-04-10 15:48:33

beautiful

time pacman -Ss pacman
.....
real 0m48.018s
user 0m0.420s
sys 0m0.780s
pacman-cage
time pacman -Ss pacman
.....
real 0m1.313s
user 0m0.332s
sys 0m0.348s

Amd Athlon XP 1500+
360MB~ ram

Thank you so much

kozaki · 2006-04-10 18:20:55

Many thanks ody

Here's the time results on an amd3200 with 1024MB RAM &
I do like that it will stuck to such speed :

Before anything :

$ time pacman -Ss pacman
real 0m14.104s
user 0m0.346s
sys 0m0.764s

After # pacman-optimize :

$ time pacman -Ss pacman
real 0m0.588s
user 0m0.270s
sys 0m0.295s

After pacman-cage :

$ time pacman -Ss pacman
real 0m0.527s
user 0m0.236s
sys 0m0.265s

I couldn't run it on my PII 350MHz, 260 MB RAM gateway which kernel doesn't have loop support

iBertus · 2006-04-10 23:47:28

I'm getting this without the above script, but would like to try it out later given the good results (can't imagine gaining much with this current setup).

real    0m0.355s
user    0m0.155s
sys    0m0.186s

Pierluigi · 2006-04-14 20:21:56

Sander wrote:

ioctl: LOOP_CLR_FD: Apparaat of bron bezig ((Device or resource busy))
mount: u moet de bestandssysteem soort aangeven ((you have to indicate the fs type))

I had the same problem, solved changing the mkfs.ext2 options:

mkfs.ext2 -O dir_index -b 1024 -m 0 -F $pacmandb

(added the "-F" flag)

L0cutus · 2006-05-15 10:51:55

seems there are problems with latest kernel update,
when i run the classic "pacman -Syu" it stay freezed, i must kill it...
loop bug ?

thanks

Arch Linux

#1 2006-04-06 17:05:37

Pacman Optimization - Part 2

#2 2006-04-06 17:26:52

Re: Pacman Optimization - Part 2

#3 2006-04-06 17:33:40

Re: Pacman Optimization - Part 2

#4 2006-04-07 16:42:49

Re: Pacman Optimization - Part 2

#5 2006-04-07 17:41:48

Re: Pacman Optimization - Part 2

#6 2006-04-07 17:52:53

Re: Pacman Optimization - Part 2

#7 2006-04-07 18:29:40

Re: Pacman Optimization - Part 2

#8 2006-04-07 19:05:48

Re: Pacman Optimization - Part 2

#9 2006-04-07 19:21:38

Re: Pacman Optimization - Part 2

#10 2006-04-07 19:41:05

Re: Pacman Optimization - Part 2

#11 2006-04-07 19:45:35

Re: Pacman Optimization - Part 2

#12 2006-04-07 20:15:52

Re: Pacman Optimization - Part 2

#13 2006-04-07 20:33:16

Re: Pacman Optimization - Part 2

#14 2006-04-07 21:11:45

Re: Pacman Optimization - Part 2

#15 2006-04-07 21:50:58

Re: Pacman Optimization - Part 2

#16 2006-04-07 21:53:03

Re: Pacman Optimization - Part 2

#17 2006-04-07 22:10:27

Re: Pacman Optimization - Part 2

#18 2006-04-08 11:09:41

Re: Pacman Optimization - Part 2

#19 2006-04-08 22:10:44

Re: Pacman Optimization - Part 2

#20 2006-04-09 15:37:39

Re: Pacman Optimization - Part 2

#21 2006-04-10 15:48:33

Re: Pacman Optimization - Part 2

#22 2006-04-10 18:20:55

Re: Pacman Optimization - Part 2

#23 2006-04-10 23:47:28

Re: Pacman Optimization - Part 2

#24 2006-04-14 20:21:56

Re: Pacman Optimization - Part 2

#25 2006-05-15 10:51:55

Re: Pacman Optimization - Part 2

Board footer