You are not logged in.

#1 2006-04-06 17:05:37

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Pacman Optimization - Part 2

So people notice I now have a decent script for this here it is:

#!/bin/bash
#
#   pacman-cage
#
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8'
dbroot="/var/lib/pacman"
pacmandb="/var/lib/pacman.db"

usage() {
        echo "pacman-cage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-cage creates a loopbacked filesystem in a contigious file."
        echo "This will give better response times when using pacman"
        echo
}

die() {
        echo "pacman-cage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

if [ "`id -u`" != 0 ]; then
        die "You must be root to cage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi
# make sure pacman.db hasnt already been made
if [ -f $pacmandb ]; then
        die "$pacmandb already exists!."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory"
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck
 
# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> creating pacman.db loopback file..."
dd if=/dev/zero of=$pacmandb bs=1M count=150 > /dev/null 2>&1

echo "==> creating ext2 -O dir_index -b 1024 -m 0 on $pacmandb..."
yes | mkfs.ext2 -O dir_index -b 1024 -m 0 $pacmandb > /dev/null 2>&1
       
echo "==> creating temporary mount point /mnt/tmp-pacman.."
mkdir /mnt/tmp-pacman

echo "==> mounting pacman.db to temporary mount point..."
mount -o loop $pacmandb /mnt/tmp-pacman

echo "==> copying pacman database to temporary mount point..."
cp -a /var/lib/pacman/. /mnt/tmp-pacman

echo "==> unmounting temporary mount point..."
umount /mnt/tmp-pacman

echo "==> removing temporary mount point..."
rmdir /mnt/tmp-pacman
       
echo "==> moving old /var/lib/pacman to /var/lib/pacman.bak..."
mv /var/lib/pacman /var/lib/pacman.bak
       
echo "==> createing new pacman db mount point @ $dbroot..."
mkdir $dbroot

echo "==> Mounting new pacman db..."
mount -o loop $pacmandb $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        umount $dbroot
        rm $pacmandb
        mv $dbroot.bak $dbroot
        die_r "integrity check FAILED, reverting to old database"
fi

echo "==> Updating /etc/fstab to reflect changes..."
echo "$pacmandb $dbroot ext2 loop,defaults 0 0" >> /etc/fstab
 
rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new
 
echo
echo "Finished.  Your pacman database has been caged!.  May the speedy pacman be with you."
echo

exit 0

Offline

#2 2006-04-06 17:26:52

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: Pacman Optimization - Part 2

ody;

To try this new program in view of the present older version already in place, what steps are advised?

I assume this program is run directly from root prompt.


Sign of the times: Navajo blanket..made in China
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#3 2006-04-06 17:33:40

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Re: Pacman Optimization - Part 2

lilsirecho wrote:

ody;

To try this new program in view of the present older version already in place, what steps are advised?

I assume this program is run directly from root prompt.

I'm just sorting out a pacman-uncage as we speak.  brb.

Offline

#4 2006-04-07 16:42:49

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Re: Pacman Optimization - Part 2

OK, heres the uncage script:

#!/bin/bash
#
#   pacman-uncage
#
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8'
dbroot="/var/lib/pacman"
tmproot="/var/lib/pacman.new"
pacmandb="/var/lib/pacman.db"

usage() {
        echo "pacman-uncage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-uncage returns your pacman db to the generic style."
        echo
}

die() {
        echo "pacman-uncage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

if [ "`id -u`" != 0 ]; then
        die "You must be root to uncage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory"
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck

# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> copying pacman.db contents back, note: the time needed to get a brew is now."
mkdir $tmproot
cp -a $dbroot/. $tmproot

echo "==> unmounting old dbroot and moving new one in"
umount $dbroot
rmdir $dbroot
mv $tmproot $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        rm -rf $dbroot
        mkdir $dbroot
        mount -a
        die_r "integrity check FAILED, reverting to old database"
fi

echo "==> Removing old pacman.db"
rm $pacmandb

rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new
            
echo
echo "Finished.  Your pacman database has been uncaged!. Welcome home."
echo "You will need to remove the old mount line from your /etc/fstab"
echo

exit 0

Offline

#5 2006-04-07 17:41:48

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: Pacman Optimization - Part 2

Am I to assume that this new uncage program applies to the original posting of pacman-cage and the newer one the subject of this thread.

To utilize this uncage program, I  would appreciate the correct entries to be made to start  this new version.

Your programs have been quite well accepted by arch users.  Very nice!!!


Sign of the times: Navajo blanket..made in China
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#6 2006-04-07 17:52:53

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

Very good !

Do not forget to build kernel with Loopback device support (BLK_DEV_LOOP)

Offline

#7 2006-04-07 18:29:40

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Re: Pacman Optimization - Part 2

lilsirecho wrote:

Am I to assume that this new uncage program applies to the original posting of pacman-cage and the newer one the subject of this thread.

To utilize this uncage program, I  would appreciate the correct entries to be made to start  this new version.

Your programs have been quite well accepted by arch users.  Very nice!!!

The uncage will work with the first version of pacman-cage back in the first thread, first post.

Offline

#8 2006-04-07 19:05:48

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

Just a little test with my old 350Mhz Celeron Notebook 64mb ram and ext3+dir_index:

before pacman-cage wrote:

# pacman-optimize
# time pacman -Ss pacman
...
real    0m58.469s
user    0m2.279s
sys    0m3.961s

after pacman-cage wrote:

# time pacman -Ss pacman
...
real    0m22.234s
user    0m2.250s
sys    0m3.181s

Amazing  8)
Thank you ody !

edit: it's an old notebook

Offline

#9 2006-04-07 19:21:38

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Re: Pacman Optimization - Part 2

wain wrote:

Just a little test with my old 350Mhz Celeron Laptop 64mb in ext3+dir_index:

before wrote:

# pacman-optimize
# time pacman -Ss pacman
...
real    0m58.469s
user    0m2.279s
sys    0m3.961s

after wrote:

# time pacman -Ss pacman
...
real    0m22.234s
user    0m2.250s
sys    0m3.181s

Amazing  8)
Thank you ody !

I think your useing Judds pacman-optimize, I didn't make that, its part of the pacman package! my process is different, try the script in the first post and see how it fairs.

Offline

#10 2006-04-07 19:41:05

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

no, I just used pacman-optimize before making the first test (58sec).

The test "after" (22sec) is made with pacman-cage !  8)

pacman-cage is even better than what you think !   :shock:

Offline

#11 2006-04-07 19:45:35

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Re: Pacman Optimization - Part 2

wain wrote:

no, I just used pacman-optimize before making the first test (58sec).

The test "after" (22sec) is made with pacman-cage !  8)

Ah right, sorry! glad its helped your celeron out! cheers for the stats aswell, nice to know it makes a real difference.

Offline

#12 2006-04-07 20:15:52

Gullible Jones
Member
Registered: 2004-12-29
Posts: 4,863

Re: Pacman Optimization - Part 2

Hey Wain, are you using XFS? Because that's the only filesystem I know of where pacman -Ss takes more than 15 seconds on a decent system.

Offline

#13 2006-04-07 20:33:16

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: Pacman Optimization - Part 2

ody;
Ran time pgm on original pacman-cage:

=================================================

[root@n6re ~]# time pacman -Ss pacman
current/pacman 2.9.8-1
    A .tar.gz based package manager with dependency support
extra/namcap 1.5.3.1-1
    A Pacman package analyzer
extra/srcpac 0.4.1-1
    The pacman from-source wrapper
community/gtkpacman 1.2-2
    A frontend written in python, using pygtk, to pacman, the archlinux package
    manager
community/jacman 0.3.1-2
    Java-based GUI front-end for pacman

real    0m0.782s
user    0m0.420s
sys     0m0.320s
[root@n6re ~]#

================================================

?????? :?:  :?:


Sign of the times: Navajo blanket..made in China
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#14 2006-04-07 21:11:45

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

Gullible Jones wrote:

Hey Wain, are you using XFS? Because that's the only filesystem I know of where pacman -Ss takes more than 15 seconds on a decent system.

Is a celeron 350Mhz with 64mb ram a decent system for you ?  lol
I'm using ext3 with dir_index. Best choice for this old notebook  wink

Offline

#15 2006-04-07 21:50:58

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: Pacman Optimization - Part 2

Weee.
I fscked up, somehow.

Does anyone know how to install packages from a given directory using pacman -S (so it'd resolve dependencies)? Command involves using ls, sed/cut/whatever and xargs, but I can't find the final solution.

Offline

#16 2006-04-07 21:53:03

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: Pacman Optimization - Part 2

Weee.
I fscked up, somehow.

Does anyone know how to install packages from a given directory using pacman -S (so it'd resolve dependencies)? Command involves using ls, sed/cut/whatever and xargs, but I can't find the final solution.

Offline

#17 2006-04-07 22:10:27

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 212
Website

Re: Pacman Optimization - Part 2

xhectorx wrote:

ok the solution was to move /var/lib/pacman.bak to /var/lib/pacman i tought the script had done this.
(pacman-cage: integrity check FAILED, reverting to old database)
well anyways..

I havnt finalised the script really.. and never thought I'd see the day Linux was built without ext2 support.. anyway atleast it created a backup, you will also have a redundant db @ /mnt/tmp-pacman that you need to remove.

I will put better sanity checking in place.

Offline

#18 2006-04-08 11:09:41

Sander
Member
Registered: 2006-02-26
Posts: 137

Re: Pacman Optimization - Part 2

I get the following when I run the script:

sander ~ > sudo exec/pacman-cage
==> md5sum'ing the old database...
==> creating pacman.db loopback file...
==> creating ext2 -O dir_index -b 1024 -m 0 on /var/lib/pacman.db...
==> creating temporary mount point /mnt/tmp-pacman..
==> mounting pacman.db to temporary mount point...
ioctl: LOOP_CLR_FD: Apparaat of bron bezig ((Device or resource busy))
mount: u moet de bestandssysteem soort aangeven ((you have to indicate the fs type))
==> copying pacman database to temporary mount point...
==> unmounting temporary mount point...
umount: /mnt/tmp-pacman: niet aangekoppeld ((not mounted))
==> removing temporary mount point...
rmdir: `/mnt/tmp-pacman': Map niet leeg ((directory not empty))
==> moving old /var/lib/pacman to /var/lib/pacman.bak...
==> createing new pacman db mount point @ /var/lib/pacman...
==> Mounting new pacman db...
ioctl: LOOP_CLR_FD: Apparaat of bron bezig ((Device or resource busy))
mount: u moet de bestandssysteem soort aangeven ((you have to indicate the fs type))
==> md5sum'ing the new database...
==> checking integrity...
umount: /var/lib/pacman: niet aangekoppeld ((not mounted))
pacman-cage: integrity check FAILED, reverting to old database

I added translations from Dutch in (())'s where necessary.

I googled for the error but all I found was about people trying to mount their initrd images as loopback devices or something =/ anyways. I'm using the 2.6.15-archck kernel. Any ideas, anyone? I'd really like to try this, as I have to run pacman-optimize almost every 2 days to get decent searching speed sad

additional note: if I try to mount an iso file (I randomly chose the arch 0.7.1 iso here) it works just fine =/


You like cheese? You like peas? You'll love cheezy peas!

Offline

#19 2006-04-08 22:10:44

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

Why not doing the same job for /var/abs/ directory ?  big_smile

Offline

#20 2006-04-09 15:37:39

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: Pacman Optimization - Part 2

ody;

On page one of this thread, there are several listings of results obtaines using ...pacman -Ss pacman.

I entered my results as well and am wondering if the test is valid due to the very stark difference in the time reported for the three elements, my values are less than a second, in case you didn't notice.

EDIT:  I unmounted the /var/lib/pacman.db and proved that it is actively running pacman when mounted.


Sign of the times: Navajo blanket..made in China
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#21 2006-04-10 15:48:33

cmf
Member
Registered: 2003-10-18
Posts: 86

Re: Pacman Optimization - Part 2

beautiful

time pacman -Ss pacman
.....
real    0m48.018s
user    0m0.420s
sys     0m0.780s

pacman-cage
time pacman -Ss pacman
.....
real    0m1.313s
user    0m0.332s
sys     0m0.348s

Amd Athlon XP 1500+
360MB~ ram

Thank you so much

Offline

#22 2006-04-10 18:20:55

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 652
Website

Re: Pacman Optimization - Part 2

Many thanks ody smile

Here's the time results on an amd3200 with 1024MB RAM &
I do like that it will stuck to such speed :

Before anything :

$ time pacman -Ss pacman
real    0m14.104s
user    0m0.346s
sys     0m0.764s

After # pacman-optimize :

$ time pacman -Ss pacman
real    0m0.588s
user    0m0.270s
sys     0m0.295s

After pacman-cage :

$ time pacman -Ss pacman
real    0m0.527s
user    0m0.236s
sys     0m0.265s

I couldn't run it on my PII 350MHz, 260 MB RAM gateway which kernel doesn't have loop support sad


Seeded this month: Arch 80 gig (and Talking~ 2, ~-anywhere 46, Apricity 16)
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) & 2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

#23 2006-04-10 23:47:28

iBertus
Member
From: Greenville, NC
Registered: 2004-11-04
Posts: 2,228

Re: Pacman Optimization - Part 2

I'm getting this without the above script, but would like to try it out later given the good results (can't imagine gaining much with this current setup).

real    0m0.355s
user    0m0.155s
sys    0m0.186s

Offline

#24 2006-04-14 20:21:56

Pierluigi
Member
Registered: 2004-04-10
Posts: 90
Website

Re: Pacman Optimization - Part 2

Sander wrote:

ioctl: LOOP_CLR_FD: Apparaat of bron bezig ((Device or resource busy))
mount: u moet de bestandssysteem soort aangeven ((you have to indicate the fs type))

I had the same problem, solved changing the mkfs.ext2 options:

mkfs.ext2 -O dir_index -b 1024 -m 0 -F $pacmandb

(added the "-F" flag)

Offline

#25 2006-05-15 10:51:55

L0cutus
Member
Registered: 2005-11-17
Posts: 24

Re: Pacman Optimization - Part 2

seems there are problems with latest kernel update,
when i run the classic "pacman -Syu" it stay freezed, i must kill it...
loop bug ?

thanks

Offline

Board footer

Powered by FluxBB