You are not logged in.

#51 2007-01-18 08:41:46

Pajaro
Member
Registered: 2004-04-21
Posts: 884

Re: Pacman Optimization - Part 2

I had to remove the cage!

The filesystem was getting corrupted every week, having to hard reboot my computer when running pacman.

I would consider updating your script to create an ext3 partition, or perform a check on every reboot.

EDIT: ops, i see the posts now. already done! tongue

Offline

#52 2007-01-18 21:11:48

Thikasabrik
Member
Registered: 2004-02-23
Posts: 92

Re: Pacman Optimization - Part 2

For those having problems with hangs who are using the beyond or CK kernels, try adding this to rc.local:

echo 33 > /proc/sys/vm/dirty_ratio

There seems to be an issue uncovered with ck-based kernels that is only triggered on some systems, and it appears to be set off by a default value of 0 for this tunable (mainline default is 40. see the ck mailing list http://bhhdoa.org.au/pipermail/ck/2007- … 6545.html).
I had problems with hangs, almost always when using pacman over a loop device, and setting this tunable as above seems to fix them.

Offline

#53 2007-01-19 17:30:00

WiLLiE
Member
From: Sweden
Registered: 2005-02-27
Posts: 135

Re: Pacman Optimization - Part 2

I had 3 hangs yesterday and removed the caged pacman.
But this might be worth a shot. Thanks. smile

Edit: still up and running. (2 days later) big_smile

Offline

#54 2007-01-22 17:22:08

sh__
Member
Registered: 2005-07-19
Posts: 272

Re: Pacman Optimization - Part 2

Thikasabrik wrote:

echo 33 > /proc/sys/vm/dirty_ratio

Been testing this for a couple of days and it seems that this prevents the hangups.

Big thanks for bringing out this info.

Offline

#55 2007-11-26 23:03:52

PDExperiment626
Member
From: Australia
Registered: 2007-04-02
Posts: 66

Re: Pacman Optimization - Part 2

While I do not use pacman-caged, I do keep my pacman.db mounted on a looped ext2 fs. I've been using my own patchset for the 2.6.22 kernel that is based around the CK patches, and I have never had a problem with my pacman.db getting corrupted. Perhaps the issue comes from the type of fs the pacman.db loop file is stored on (I use JFS). Anyway, that might be something to play around with.

I do actually have a question though. While I am getting good performance with a pacman.db caged in ext2, I am wondering if there isn't a better (i.e. faster) fs to cage this in? I was kind of surprised looking at an Arch interview with some of the devs saying that pacman had performance issues with Reiserfs. I know it's a journaled fs, but I thought Rfs was optimized for working with many small files like in the pacman db? Anyway, I figure ext2 is the fastest due to it not being journaled and the way it handles disk caching; but it can't hurt to ask smile.


... and for a time, it was good...

Offline

#56 2007-11-27 02:22:11

delphiki
Member
Registered: 2007-11-17
Posts: 66
Website

Re: Pacman Optimization - Part 2

@PDExperiment626: Going along with your idea, couldn't you also use a loopback mounted iso9660 filesystem like a CD ISO image? In other words you make an iso image of the pacman-cage and mount it using the loopback option?

Offline

#57 2008-05-09 06:51:25

el mariachi
Member
Registered: 2007-11-30
Posts: 595

Re: Pacman Optimization - Part 2

continuing PDExperiments626 line of thought, is there a way to use JFS instead of ext2?

Offline

#58 2008-06-14 16:00:39

turkystuffing
Member
Registered: 2008-05-27
Posts: 3

Re: Pacman Optimization - Part 2

I went from:
time yaourt -Ss

real    0m1.290s
user    0m0.727s
sys    0m0.543s

to:
time yaourt -Ss

real    0m0.293s
user    0m0.177s
sys    0m0.100s

Which is pretty impressive IMO. I'm running a 2Ghz Dual core AMD cpu with 2gb of ram, XFS file system.

Offline

#59 2008-06-14 19:29:22

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: Pacman Optimization - Part 2

Stythys wrote:

==> md5sum'ing the old database...
==> creating pacman.db loopback file...
==> creating ext2 -O dir_index -b 1024 -m 0 on /var/lib/pacman.db...
==> creating temporary mount point /mnt/tmp-pacman..
==> mounting pacman.db to temporary mount point...
mount: could not find any device /dev/loop#

now my database is gone and pacman thinks I have nothing installed =/
how do I restore it?

Maybe you should fix the script so that :
1) it does a backup
2) it checks for the return value of mount and exits cleanly if it didn't go fine
3) it loads the loop module


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#60 2008-07-03 07:41:12

ody
Member
From: Manchester, UK
Registered: 2002-08-12
Posts: 216
Website

Re: Pacman Optimization - Part 2

Hi Shining

I wrote the script in a very slapstick manner thinking people would have a read of it before running.  Anyway, the old database is moved to: /var/lib/pacman.bak so you can simply move it back to /var/lib/pacman and your be back to where you started.  I'm amazed so many people seem to have missed this.. very sorry hmm

Offline

#61 2008-08-25 17:29:06

NicolasDiaz
Member
Registered: 2008-08-25
Posts: 5

Re: Pacman Optimization - Part 2

Hi

I touched the script created by Andrew, I extend the checks for loopback support, remplace some values for var's,
and, when the size of loop file is created, the size is determinated using as reference /var/lib/pacman.

Now is posible to change the fs used in $pacmandb, just setting a variable, but some tweaks are needed in /etc/rc.d/pacmandb and
pacman-uncage scripts.

Bye smile

#!/bin/bash
#
#   pacman-cage
#
#   Copyright (c) 2002-2006 by Pablo Nicolas Diaz <pablonicolas.diaz@gmail.com>
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8.3'
dbroot="/var/lib/pacman/"
pacmandb="/var/lib/pacman-X.db"

# filesystem used in pacmandb
filesystem="ext2"
# commandline needed for mkfs.$filesystem and dd
BlockSize="1024"
Options="-F -O dir_index -b $BlockSize -m 0"
# temporal mountpoint 
pacman_tmp_mnt_point=/mnt/tmp-pacman

usage() {
        echo "pacman-cage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-cage creates a loopbacked filesystem in a contigious file."
        echo "This will give better response times when using pacman"
        echo
}

die() {
        echo "pacman-cage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

# test if filesystem support is present
if ! grep $filesystem /proc/filesystems > /dev/null; then
    die "filesystem $filesystem is not supported!" 
fi

# added by wain: make sure kernel support loop device
if ! zcat /proc/config.gz | grep "BLK_DEV_LOOP=\(y\|m\)" >/dev/null; then
       die "please enable loop device support in kernel (BLK_DEV_LOOP=y)"
elif [ -d /dev/loop/ ]; then 
       die "You must insert the loopback module!"
fi

if [ "`id -u`" != 0 ]; then
       die "You must be root to cage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi
# make sure pacman.db hasnt already been made
if [ -f $pacmandb ]; then
        die "$pacmandb already exists!."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory"
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck
 
# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> creating pacman.db loopback file..."
CountSize=$(( $(du -s --block-size=$BlockSize $dbroot | cut -f 1) * 5/3 ))

# echo $CountSize
dd if=/dev/zero of=$pacmandb bs=$BlockSize count=$CountSize > /dev/null 2>&1

echo "==> creating $filesystem $Options on $pacmandb ..."
mkfs.$filesystem $Options $pacmandb > /dev/null 2>&1
       
echo "==> creating temporary mount point $pacman_tmp_mnt_point.."
mkdir $pacman_tmp_mnt_point;
dir $pacman_tmp_mnt_point

echo "==> mounting pacman.db to temporary mount point..."
mount -t $filesystem -o loop $pacmandb $pacman_tmp_mnt_point

echo "==> copying pacman database to temporary mount point..."
cp -a $dbroot. $pacman_tmp_mnt_point

echo "==> unmounting temporary mount point..."
umount $pacman_tmp_mnt_point

echo "==> removing temporary mount point..."
rmdir $pacman_tmp_mnt_point
       
echo "==> moving old /var/lib/pacman to /var/lib/pacman.bak..."
mv $dbroot /var/lib/pacman.bak
       
echo "==> createing new pacman db mount point @ $dbroot..."
mkdir $dbroot

echo "==> Mounting new pacman db..."
mount -t $filesystem -o loop $pacmandb $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        umount $dbroot
        rm $pacmandb
        mv $dbroot.bak $dbroot
        die_r "integrity check FAILED, reverting to old database"
fi

rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new

# Replaced by wain: mount is now in /etc/rc.d/pacmandb daemon
#echo "==> Updating /etc/fstab to reflect changes..."
#echo "$pacmandb $dbroot $filesystem loop,defaults 0 0" >> /etc/fstab
 
echo
echo "Finished.  Your pacman database has been caged!.  May the speedy pacman be with you."
echo

exit 0

Offline

#62 2008-08-25 19:00:56

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

Nice smile

Just 2 things:
- /tmp/pacman.lck have to be replaced by $(LC_ALL=C pacman -v | grep 'Lock File' | awk '{print $4}') or default /var/lib/pacman/db.lck
- dbroot can be replaced the same way by $(LC_ALL=C pacman -v | grep 'DB Path' | awk '{print $4}')

Do you feel best performance with others options or different filesystem ?

Offline

#63 2008-08-30 16:27:21

NicolasDiaz
Member
Registered: 2008-08-25
Posts: 5

Re: Pacman Optimization - Part 2

Nice ...

I'm trying to use reiserfs, and in my x86 (celeron D 320 (2.4 GHz, 533Mhz FSB ,256Kb cache , 512 Mb Ram and a old 40 Gb PATA disk)) is better reiserfs that ext2 or ext3, but in a x86_64 with 2GB Ram and Intel(R) Core(TM)2 CPU 4300  @ 1.80GHz with 2 cores, the speedup is better with ext2 or ext3.

in the next post, i will post the results,because ssh is not working well in my x86 , I'm far (150Km) from my machine

Offline

#64 2009-04-06 09:11:31

Fackamato
Member
Registered: 2006-03-31
Posts: 579

Re: Pacman Optimization - Part 2

Is anyone using this?

Offline

#65 2009-04-06 12:00:03

Daenyth
Forum Fellow
From: Boston, MA
Registered: 2008-02-24
Posts: 1,244

Re: Pacman Optimization - Part 2

pacman-optimize is included in the pacman package

Offline

#66 2009-04-06 13:11:12

Fackamato
Member
Registered: 2006-03-31
Posts: 579

Re: Pacman Optimization - Part 2

Daenyth wrote:

pacman-optimize is included in the pacman package

Uh yeah but RAM is quite much faster than HDD still, isn't it? wink

Offline

#67 2009-06-06 19:12:14

moz
Member
Registered: 2009-01-20
Posts: 17

Re: Pacman Optimization - Part 2

Hi,
Ive had a strange problem. The filesystem seems to get full after a while, atleast so it seems cos i cant write anymore into it even though theres about 90 megs free. I checked with #tune2fs -l /var/lib/pacman.db and i noticed theres no more free inodes...
If anyone can shed some light on this cos i dont know what to make of it. I had to make a new fs of 300Mb this time and copy stuff over.

tune2fs 1.41.5 (23-Apr-2009)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          7fe88983-7117-4c20-9225-7250b60b0a6e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              38456
Block count:              153600
Reserved block count:     0
Free blocks:              92936
Free inodes:              1
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2024
Inode blocks per group:   253
Filesystem created:       Sat May 23 08:53:54 2009
Last mount time:          Sat Jun  6 19:52:33 2009
Last write time:          Sat Jun  6 20:16:07 2009
Mount count:              3
Maximum mount count:      21
Last checked:             Sat Jun  6 19:06:26 2009
Check interval:           15552000 (6 months)
Next check after:         Thu Dec  3 18:06:26 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Default directory hash:   half_md4
Directory Hash Seed:      0337409d-0db8-4104-b4d0-eb3f39c4f31b

Offline

#68 2009-06-07 10:14:30

Themaister
Member
From: Trondheim, Norway
Registered: 2008-07-21
Posts: 652
Website

Re: Pacman Optimization - Part 2

Wow o.O This is awesome. I don't quite understand what this script does though. Is it something like mounting the database in a tmpfs?

Offline

#69 2009-06-07 18:25:42

Thikasabrik
Member
Registered: 2004-02-23
Posts: 92

Re: Pacman Optimization - Part 2

moz wrote:

Hi,
Ive had a strange problem. The filesystem seems to get full after a while, atleast so it seems cos i cant write anymore into it even though theres about 90 megs free. I checked with #tune2fs -l /var/lib/pacman.db and i noticed theres no more free inodes...
If anyone can shed some light on this cos i dont know what to make of it. I had to make a new fs of 300Mb this time and copy stuff over.

I recommend modifying the script so the fs creation is like this:

mkfs.ext2 -O dir_index -b 1024 -i 1024 -m 0

The key difference being the bytes-per-inode parameter 'i'. My db file is 30mb with these options.

Offline

#70 2009-07-17 00:27:04

DarkMind
Member
Registered: 2005-02-22
Posts: 68

Re: Pacman Optimization - Part 2

very fast, amazing yikes

Offline

#71 2009-08-20 22:07:28

ruscook
Member
From: Sydney Australia
Registered: 2007-08-27
Posts: 105
Website

Re: Pacman Optimization - Part 2

I've just decided to enable pacman-cage and yes it is fast! Obviously I'm also concerned that I don't lose data using the loopback mount.

What's I've done is add an rsync backup on successful mounting of the loopback f/s on booting. To do this I modified the rc.local script already posted here (sorry can't remember who at the moment).

Here's the modded script.

#!/bin/bash
#
# /etc/rc.local: Local multi-user startup script.
#
. /etc/rc.conf
. /etc/rc.d/functions


# Check and mount pacman database
stat_busy "Mounting the pacman database"
/sbin/e2fsck -p /var/lib/pacman-X.db
if [ $? -gt 1 ]; then
   stat_fail
   echo
   echo "WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED"
   echo
else
   mount /var/lib/pacman
   if [ $? -gt 0 ]; then
      stat_fail
      echo
      echo "WARNING: MOUNTING PACMAN DATABASE FAILED"
      echo
   else
      stat_done
    stat_busy "Backing up the pacman database"    
    rsync -Caxq --delete-after --exclude "lost+found/" /var/lib/pacman/  /var/lib/pacman-backup/
      stat_done
   fi
fi

As it's rsync and only changed files, it's pretty quick so I'm not worried about speed. I can also cron this if I wanted to ensure backups between boots.

Can any tell me if there's any hidden gotcha's (issues) with this idea or is it safe to do the rsync on each boot?
Thanks
Russ

Offline

#72 2009-08-21 07:04:03

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

ruscook wrote:

Can any tell me if there's any hidden gotcha's (issues) with this idea or is it safe to do the rsync on each boot?

I don't know but you just have to backup /var/lib/pacman/local/. It's faster.

I use pacman-cage for many years without any problem.

Offline

#73 2009-08-24 11:13:07

ruscook
Member
From: Sydney Australia
Registered: 2007-08-27
Posts: 105
Website

Re: Pacman Optimization - Part 2

wain wrote:
ruscook wrote:

Can any tell me if there's any hidden gotcha's (issues) with this idea or is it safe to do the rsync on each boot?

I don't know but you just have to backup /var/lib/pacman/local/. It's faster.

I use pacman-cage for many years without any problem.

Thanks Wain. Why only local/ not sync/ as well?

Russ

Offline

#74 2009-08-24 11:25:48

wain
Member
From: France
Registered: 2005-05-01
Posts: 289
Website

Re: Pacman Optimization - Part 2

ruscook wrote:

Thanks Wain. Why only local/ not sync/ as well?

Because "sync" directory is created by pacman -Sy and don't contain any personnal info

Offline

#75 2009-08-27 08:29:53

ruscook
Member
From: Sydney Australia
Registered: 2007-08-27
Posts: 105
Website

Re: Pacman Optimization - Part 2

Ah even better! Thanks, will adjust my script.
Russ

Offline

Board footer

Powered by FluxBB