Pacman Optimization

jaboua · 2006-04-03 15:23:32

hellwoofa wrote:

I created the ext3-fs on my root with the pacman setup, is the block size of 4k normal? You said 1k, which looks more sensible, is the default.. strange.

AFAIK, bigger block-sizes can increase performance and fragment less, but use more diskspace...

lilsirecho · 2006-04-03 15:32:42

ody;

After upgrade to 2.6.16 kernel, everything performing normally with the pacman change.

See a great interest in your program and many seem to like it.

Best to you...

EDIT: In kdiskfree, opening in file manager the /var/lib,pacman.db files provides access to all and also a listing of the selected repository in alphabetical order in the left margin as well as a display of the packages in the selected repo. Handy!

ody · 2006-04-03 19:15:37

lilsirecho wrote:

ody;
After upgrade to 2.6.16 kernel, everything performing normally with the pacman change.
See a great interest in your program and many seem to like it.
Best to you...
EDIT: In kdiskfree, opening in file manager the /var/lib,pacman.db files provides access to all and also a listing of the selected repository in alphabetical order in the left margin as well as a display of the packages in the selected repo. Handy!

I'm glad you like lilsirecho. The slowness of pacman has always been a pain in my ass, finaly instead of bitching about it I came up with a solution that I'm happy (but feel pretty dumb about because it's so simple) with and others seem to like to. Maybe Judd will intergrate it as a pacman util for all to enjoy

sud_crow · 2006-04-04 03:16:09

Hi,
I was wondering after seeing the filesystem for the pacman.db file being ext3, and as you are all making comparisons, it would be nice to have it with ReiserFS... is this possible?
I read lots of times that reiserfs has real nice performance with small files, here are a couple of benchs i just found googling (although the second its kind of confussing ):

http://navindra.blogspot.com/2004/10/kd … ilure.html
http://www.gurulabs.com/goodies/ext3_vs_reiserfs-1.php (based on 2002!!)

Anyway, the best way to know would be to try, i would do it myself if i knew how

iphitus · 2006-04-04 08:25:37

sud_crow wrote:

Hi,
I was wondering after seeing the filesystem for the pacman.db file being ext3, and as you are all making comparisons, it would be nice to have it with ReiserFS... is this possible?
I read lots of times that reiserfs has real nice performance with small files, here are a couple of benchs i just found googling (although the second its kind of confussing ):
http://navindra.blogspot.com/2004/10/kd … ilure.html
http://www.gurulabs.com/goodies/ext3_vs_reiserfs-1.php (based on 2002!!)
Anyway, the best way to know would be to try, i would do it myself if i knew how

*shrug* pimp your machine however you like. I doubt you'd see much difference between the two in this case.

Benchmarks are meaningless, as not only are they old, but would most likely be based on entirely different setups.

James

user · 2006-04-04 10:43:32

sud_crow wrote:

Hi,
it would be nice to have it with ReiserFS... is this possible?

Hi,

my computer(arch 0.7.1) has reiserfs 3. the installation process little confusing, but you can do it.

brain0 · 2006-04-04 12:29:05

ody: Four suggestions:

1) WIkify this
2) Use tune2fs to omit regular file system checks (or maybe not)
3) I recently read that using journaling on a loopback file is pretty much pointless, as you cannot be sure that data is written as you want it to be written. Use ext2 instead.
4) Adding -O dir_index may prove useful.

syamajala: Using a fuse filesystem as a backend for a database would be possible. Reading it would be easy in this case, but writing would be more tricky. Of course, this would really increase performance.

brain0 · 2006-04-04 12:50:24

Ok, I made some further improvements:

Do not create the image with dd, but rather with a tool like qemu-img (from the qemu package). The advantage is that there is no data written to the file - meaning you have a 100MB file that only takes a few bytes of hard disk space (which is possible with ext2/3 because it supports "holes" in files). Maybe there are other tools that do the same, but I don't know which.

[root@architect ~]# qemu-img create -f raw pacman.db 100M
Formating 'pacman.db', fmt=raw, size=102400 kB
[root@architect ~]# ls -lhF pacman.db 
-rw-r--r--  1 root root 100M 2006-04-04 14:39 pacman.db
[root@architect ~]# du -h pacman.db 
0       pacman.db

Now, create the filesystem (I used ext2, see my last posting) with 1024 block size and dir_index (dir_index for speed, 1024 for size):

[root@architect ~]# mke2fs -b 1024 -L pacman -m 0 -O dir_index pacman.db 
mke2fs 1.38 (30-Jun-2005)
pacman.db is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=pacman
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
25688 inodes, 102400 blocks
0 blocks (0.00%) reserved for the super user
First data block=1
13 block groups
8192 blocks per group, 8192 fragments per group
1976 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729

Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 27 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@architect ~]# tune2fs -c 0 -i 0 pacman.db 
tune2fs 1.38 (30-Jun-2005)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds

Now, copy the pacman db

[root@architect ~]# mount -o loop pacman.db /mnt/other/
[root@architect ~]# cd /mnt/other/
[root@architect other]# cp -a /var/lib/pacman/* .

The 1024 Byte blocksize decreased my pacman database size from 87MB to 16MB (checked it with du). A 512Byte blocksize would decrease it even more, but ext2 does not support that.

Now, the pacman db is in a single file, which is quite small:

[root@architect ~]# ls -lhF pacman.db 
-rw-r--r--  1 root root 100M 2006-04-04 14:40 pacman.db
[root@architect ~]# du -h pacman.db 
19M     pacman.db

19MB for my pacman db instead of 87MB before. I didn't test it with pacman yet, but I guess the speed increase is significant.

ody · 2006-04-04 17:01:10

Cheers for the insight brain0, I've decided againt the use of qemu-img as that itself would seem to fragment as it grew and also a fs check isnt the end of the world now and again. I have used you ext2 and dir_index ideas though cheers!

This is what I have now, no one use it yet unless you know what your doing, it works for me but I cant be sure it will for you.

#!/bin/bash
#
#   pacman-cage
#
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8'
dbroot="/var/lib/pacman"
pacmandb="/var/lib/pacman.db"

usage() {
        echo "pacman-cage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-cage creates a loopbacked filesystem in a contigious file."
        echo "This will give better response times when using pacman"
        echo
}

die() {
        echo "pacman-cage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

if [ "`id -u`" != 0 ]; then
        die "You must be root to cage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi
# make sure pacman.db hasnt already been made
if [ -f $pacmandb ]; then
        die "$pacmandb already exists!."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory" 
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck
 
# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> creating pacman.db loopback file..."
dd if=/dev/zero of=$pacmandb bs=1M count=150 > /dev/null 2>&1

echo "==> creating ext2 -O dir_index -b 1024 on $pacmandb..."
yes | mkfs.ext2 -O dir_index -b 1024 $pacmandb > /dev/null 2>&1
        
echo "==> creating temporary mount point /mnt/tmp-pacman.."
mkdir /mnt/tmp-pacman

echo "==> mounting pacman.db to temporary mount point..."
mount -o loop $pacmandb /mnt/tmp-pacman

echo "==> copying pacman database to temporary mount point..."
cp -a /var/lib/pacman/. /mnt/tmp-pacman

echo "==> unmounting temporary mount point..."
umount /mnt/tmp-pacman

echo "==> removing temporary mount point..."
rmdir /mnt/tmp-pacman
        
echo "==> moving old /var/lib/pacman to /var/lib/pacman.bak..."
mv /var/lib/pacman /var/lib/pacman.bak
        
echo "==> createing new pacman db mount point @ $dbroot..."
mkdir $dbroot

echo "==> Mounting new pacman db..."
mount -o loop $pacmandb $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        umount $dbroot
        rm $pacmandb
        mv $dbroot.bak $dbroot
        die_r "integrity check FAILED, reverting to old database"
fi

echo "==> Updating /etc/fstab to reflect changes..."
echo "$pacmandb $dbroot ext2 loop,defaults 0 0" >> /etc/fstab
 
rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new
  
echo
echo "Finished.  Your pacman database has been caged!.  May the speedy pacman be with you."
echo

exit 0

brain0 · 2006-04-04 17:31:10

ody wrote:

Cheers for the insight brain0, I've decided againt the use of qemu-img as that itself would seem to fragment as it grew and also a fs check isnt the end of the world now and again. I have used you ext2 and dir_index ideas though cheers!

You should still decrease the blocksize, since that will save a lot of space (You won't loose speed afaics, as a single pacman file is most likely < 300 Byte).

About qemu-img (or any similar tool), I think it is still worth more thought. Filesystems like ext2 are designed to avoid fragmentation. That may fail with large directories containing many small files, but may be more successful with one file (especially that a ~30MB file can and probably will be read into memory very quickly, at least much quicker than scanning directories for single files all over a big partition).

Maybe there is a good way to benchmark this (although you will need a way to clear linux's caches in memory).

ody · 2006-04-04 17:42:06

I did decrease it explicitly to 1k, forgot to mention. However default block sizes are 1k anyway with small filesytems like this.

vidal · 2006-04-04 19:03:20

Why not just use a real, separate partition for /var/lib/pacman?

ody · 2006-04-04 19:31:01

vidal wrote:

Why not just use a real, separate partition for /var/lib/pacman?

It wouldnt work for archers that already have there system setup, unless they use lvm I guess.. I used loop back for its flexibility.

brain0 · 2006-04-04 20:54:27

ody wrote:

I did decrease it explicitly to 1k, forgot to mention. However default block sizes are 1k anyway with small filesytems like this.

[code]mkfs.ext2 -O dir_index -s 1 $pacmandb[/code[

You did not. Unless of course, the -s option does this - it is not mentioned in the manpage though. The -b option determines the block size. I wouldn't rely on defaults, because they tend to change.

Otherwise, nice job, a shame that I didn't come up with that idea.

Did you already wikify this?

sud_crow · 2006-04-05 03:10:59

iphitus wrote:

sud_crow wrote:
Hi,
I was wondering after seeing the filesystem for the pacman.db file being ext3, and as you are all making comparisons, it would be nice to have it with ReiserFS... is this possible?
I read lots of times that reiserfs has real nice performance with small files, here are a couple of benchs i just found googling (although the second its kind of confussing ):
http://navindra.blogspot.com/2004/10/kd … ilure.html
http://www.gurulabs.com/goodies/ext3_vs_reiserfs-1.php (based on 2002!!)
Anyway, the best way to know would be to try, i would do it myself if i knew how
*shrug* pimp your machine however you like. I doubt you'd see much difference between the two in this case.
Benchmarks are meaningless, as not only are they old, but would most likely be based on entirely different setups.
James

Hi james,
I was thinking in the pacman.db filesystem, not in my machine filesystem, i already have reiserfs as my main, and as this is about getting the best performance for pacman, i thought it would be nice to try that.

I dont think performance should vary from system (as in PC) to system, i mean, if the database is in reiserfs and works better, it should do it in almost every Arch setup, what could change that?

Any way, it was just a thought

ody · 2006-04-05 07:06:41

brain0 wrote:

ody wrote:
I did decrease it explicitly to 1k, forgot to mention. However default block sizes are 1k anyway with small filesytems like this.
[code]mkfs.ext2 -O dir_index -s 1 $pacmandb[/code[
You did not. Unless of course, the -s option does this - it is not mentioned in the manpage though. The -b option determines the block size. I wouldn't rely on defaults, because they tend to change.
Otherwise, nice job, a shame that I didn't come up with that idea.
Did you already wikify this?

Your right I didnt, I was using the flag used for vfat.. ah well. I've not wikified this yet no.

Sander · 2006-04-05 08:52:55

I'm tempted to try this... how risky is it exactly? I mean, if I backup the db, there shouldn't be any risk... b0rking my db is the worst it will do... right?

ody · 2006-04-05 08:58:31

Sander wrote:

I'm tempted to try this... how risky is it exactly? I mean, if I backup the db, there shouldn't be any risk... b0rking my db is the worst it will do... right?

I've just ran it on my work machine and my collegues with no problems. It will backup your pacman db to /var/lib/pacman.bak anyway so if the unlikely did happen you can role back.

Sander · 2006-04-05 09:01:38

I'll try it on my laptop first, been experimenting with that a lot lately

ody · 2006-04-06 09:07:09

Has anyone had any problems with pacman-cage? if not anyone know the best place to put this for others to use?

cheers

lilsirecho · 2006-04-06 14:28:37

ody;

Haven't tried the second method for the pacman system you posted.

As reported, the expected block size seems not to be attained.

Any advice on whether there is enough improvement to consider trying the later posted pacman-cage?

ody · 2006-04-06 14:33:57

lilsirecho wrote:

ody;
Haven't tried the second method for the pacman system you posted.
As reported, the expected block size seems not to be attained.
Any advice on whether there is enough improvement to consider trying the later posted pacman-cage?

Not really, the only changes in the second method where changing to ext2 and using dir_index and explicity setting block size to 1k, oh and it has a more informative setup.

IceRAM · 2006-04-06 16:34:37

ody wrote:

So anyway the script below will do what I have done for you but make sure you increase the count= on the dd line to come into line with what you think you will need for your pacman db, run "du -sh /var/lib/pacman" to get an idea of how much space you will need (I would say set it to atleast 150) when creating the loopback file with dd. Now the loopback file itself may be created fragmented so the best time to do this would be during a fresh install, but I have noticed a massive increase in speed and my root is years old.

-rw-r--r--   1 root   root     50M Jun 19  2005 pacman.db.backup

I have been using this solution for... hmm... almost a year (I have even used the same name). I wanted to publish it together with some tests, but never had the time to do so and I also forgot about it.

/var/lib/pacman.db
              ext2     49M   21M   26M  45% /var/lib/pacman

Differences
* I made it ext2, not ext3, because I considered I didn't need a journal for it - don't remember exactly why.
* I'm almost sure I tried to get the smallest block size
* also, I've tried to minimize the number of reserved blocks for the superuser

Well, these days, during the latest updates, it seems the filesystem came to its end. I don't remember exactly the parameters I used when making the filesystem - and I don't know how to check them.

Notice that I made it 50MB big. Unfortunately, even though the usage is as above, pacman often returns "out of space" and has problems writing package info to disk.

I'm about to make the filesystem 150MB big. I hope it will work. Even so, the situation described above is a bit strange.

IceRAM · 2006-04-06 16:49:41

another strange thing:

/var/lib/pacman.db
              ext2     49M   21M   26M  45% /var/lib/pacman
/var/lib/pacman.db.2
              ext3    146M   25M  121M  17% /mnt/tmp-pacman

Notice the extra MB used on the 2nd db, even thout it has the exact structure as the first one. It is highly unlikely IMO for that space to be lost because of the journal. The block size is 1KB, which seems the minimum acceptable for both ext2 and ext3.

This time I've created the loop filesystem with:

mkfs.ext3 -m 0 /var/lib/pacman.db.2

-m 0 = no space reserverd for the superuser

ody · 2006-04-06 17:08:05

Hey IceRAM, this is something a lot of people have thought about or even implemented, I've just stuck it into an easy to use script, now in the new thread "Pacman Optimization - Part 2", I have included your "-m 0".

The newer version uses ext2 (this was done a few posts back, suggested by brain0).

Arch Linux

#26 2006-04-03 15:23:32

Re: Pacman Optimization

#27 2006-04-03 15:32:42

Re: Pacman Optimization

#28 2006-04-03 19:15:37

Re: Pacman Optimization

#29 2006-04-04 03:16:09

Re: Pacman Optimization

#30 2006-04-04 08:25:37

Re: Pacman Optimization

#31 2006-04-04 10:43:32

Re: Pacman Optimization

#32 2006-04-04 12:29:05

Re: Pacman Optimization

#33 2006-04-04 12:50:24

Re: Pacman Optimization

#34 2006-04-04 17:01:10

Re: Pacman Optimization

#35 2006-04-04 17:31:10

Re: Pacman Optimization

#36 2006-04-04 17:42:06

Re: Pacman Optimization

#37 2006-04-04 19:03:20

Re: Pacman Optimization

#38 2006-04-04 19:31:01

Re: Pacman Optimization

#39 2006-04-04 20:54:27

Re: Pacman Optimization

#40 2006-04-05 03:10:59

Re: Pacman Optimization

#41 2006-04-05 07:06:41

Re: Pacman Optimization

#42 2006-04-05 08:52:55

Re: Pacman Optimization

#43 2006-04-05 08:58:31

Re: Pacman Optimization

#44 2006-04-05 09:01:38

Re: Pacman Optimization

#45 2006-04-06 09:07:09

Re: Pacman Optimization

#46 2006-04-06 14:28:37

Re: Pacman Optimization

#47 2006-04-06 14:33:57

Re: Pacman Optimization

#48 2006-04-06 16:34:37

Re: Pacman Optimization

#49 2006-04-06 16:49:41

Re: Pacman Optimization

#50 2006-04-06 17:08:05

Re: Pacman Optimization

Board footer