Pacman Optimization - Part 2

Pajaro · 2007-01-18 08:41:46

I had to remove the cage!

The filesystem was getting corrupted every week, having to hard reboot my computer when running pacman.

I would consider updating your script to create an ext3 partition, or perform a check on every reboot.

EDIT: ops, i see the posts now. already done!

Thikasabrik · 2007-01-18 21:11:48

For those having problems with hangs who are using the beyond or CK kernels, try adding this to rc.local:

echo 33 > /proc/sys/vm/dirty_ratio

There seems to be an issue uncovered with ck-based kernels that is only triggered on some systems, and it appears to be set off by a default value of 0 for this tunable (mainline default is 40. see the ck mailing list http://bhhdoa.org.au/pipermail/ck/2007- … 6545.html).
I had problems with hangs, almost always when using pacman over a loop device, and setting this tunable as above seems to fix them.

WiLLiE · 2007-01-19 17:30:00

I had 3 hangs yesterday and removed the caged pacman.
But this might be worth a shot. Thanks.

Edit: still up and running. (2 days later)

sh__ · 2007-01-22 17:22:08

Thikasabrik wrote:

echo 33 > /proc/sys/vm/dirty_ratio

Been testing this for a couple of days and it seems that this prevents the hangups.

Big thanks for bringing out this info.

PDExperiment626 · 2007-11-26 23:03:52

While I do not use pacman-caged, I do keep my pacman.db mounted on a looped ext2 fs. I've been using my own patchset for the 2.6.22 kernel that is based around the CK patches, and I have never had a problem with my pacman.db getting corrupted. Perhaps the issue comes from the type of fs the pacman.db loop file is stored on (I use JFS). Anyway, that might be something to play around with.

I do actually have a question though. While I am getting good performance with a pacman.db caged in ext2, I am wondering if there isn't a better (i.e. faster) fs to cage this in? I was kind of surprised looking at an Arch interview with some of the devs saying that pacman had performance issues with Reiserfs. I know it's a journaled fs, but I thought Rfs was optimized for working with many small files like in the pacman db? Anyway, I figure ext2 is the fastest due to it not being journaled and the way it handles disk caching; but it can't hurt to ask .

delphiki · 2007-11-27 02:22:11

@PDExperiment626: Going along with your idea, couldn't you also use a loopback mounted iso9660 filesystem like a CD ISO image? In other words you make an iso image of the pacman-cage and mount it using the loopback option?

el mariachi · 2008-05-09 06:51:25

continuing PDExperiments626 line of thought, is there a way to use JFS instead of ext2?

turkystuffing · 2008-06-14 16:00:39

I went from:
time yaourt -Ss

real 0m1.290s
user 0m0.727s
sys 0m0.543s

to:
time yaourt -Ss

real 0m0.293s
user 0m0.177s
sys 0m0.100s

Which is pretty impressive IMO. I'm running a 2Ghz Dual core AMD cpu with 2gb of ram, XFS file system.

shining · 2008-06-14 19:29:22

Stythys wrote:

==> md5sum'ing the old database...
==> creating pacman.db loopback file...
==> creating ext2 -O dir_index -b 1024 -m 0 on /var/lib/pacman.db...
==> creating temporary mount point /mnt/tmp-pacman..
==> mounting pacman.db to temporary mount point...
mount: could not find any device /dev/loop#
now my database is gone and pacman thinks I have nothing installed =/
how do I restore it?

Maybe you should fix the script so that :
1) it does a backup
2) it checks for the return value of mount and exits cleanly if it didn't go fine
3) it loads the loop module

ody · 2008-07-03 07:41:12

Hi Shining

I wrote the script in a very slapstick manner thinking people would have a read of it before running. Anyway, the old database is moved to: /var/lib/pacman.bak so you can simply move it back to /var/lib/pacman and your be back to where you started. I'm amazed so many people seem to have missed this.. very sorry

NicolasDiaz · 2008-08-25 17:29:06

Hi

I touched the script created by Andrew, I extend the checks for loopback support, remplace some values for var's,
and, when the size of loop file is created, the size is determinated using as reference /var/lib/pacman.

Now is posible to change the fs used in $pacmandb, just setting a variable, but some tweaks are needed in /etc/rc.d/pacmandb and
pacman-uncage scripts.

Bye

#!/bin/bash
#
#   pacman-cage
#
#   Copyright (c) 2002-2006 by Pablo Nicolas Diaz <pablonicolas.diaz@gmail.com>
#   Copyright (c) 2002-2006 by Andrew Rose <rose.andrew@gmail.com>
#   I used Judds pacman-optimise as a framework.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
#   USA.
#

myver='2.9.8.3'
dbroot="/var/lib/pacman/"
pacmandb="/var/lib/pacman-X.db"

# filesystem used in pacmandb
filesystem="ext2"
# commandline needed for mkfs.$filesystem and dd
BlockSize="1024"
Options="-F -O dir_index -b $BlockSize -m 0"
# temporal mountpoint 
pacman_tmp_mnt_point=/mnt/tmp-pacman

usage() {
        echo "pacman-cage $myver"
        echo "usage: $0 [pacman_db_root]"
        echo
        echo "pacman-cage creates a loopbacked filesystem in a contigious file."
        echo "This will give better response times when using pacman"
        echo
}

die() {
        echo "pacman-cage: $*" >&2
        exit 1
}

die_r() {
        rm -f /tmp/pacman.lck
        die $*
}

if [ "$1" != "" ]; then
        if [ "$1" = "-h" -o "$1" = "--help" ]; then
                usage
                exit 0
        fi
        dbroot=$1
fi

# test if filesystem support is present
if ! grep $filesystem /proc/filesystems > /dev/null; then
    die "filesystem $filesystem is not supported!" 
fi

# added by wain: make sure kernel support loop device
if ! zcat /proc/config.gz | grep "BLK_DEV_LOOP=\(y\|m\)" >/dev/null; then
       die "please enable loop device support in kernel (BLK_DEV_LOOP=y)"
elif [ -d /dev/loop/ ]; then 
       die "You must insert the loopback module!"
fi

if [ "`id -u`" != 0 ]; then
       die "You must be root to cage the database"
fi

# make sure pacman isn't running
if [ -f /tmp/pacman.lck ]; then
        die "Pacman lockfile was found.  Cannot run while pacman is running."
fi
# make sure pacman.db hasnt already been made
if [ -f $pacmandb ]; then
        die "$pacmandb already exists!."
fi

if [ ! -d $dbroot ]; then
        die "$dbroot does not exist or is not a directory"
fi

# don't let pacman run while we do this
touch /tmp/pacman.lck
 
# step 1: sum the old db
echo "==> md5sum'ing the old database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.old

echo "==> creating pacman.db loopback file..."
CountSize=$(( $(du -s --block-size=$BlockSize $dbroot | cut -f 1) * 5/3 ))

# echo $CountSize
dd if=/dev/zero of=$pacmandb bs=$BlockSize count=$CountSize > /dev/null 2>&1

echo "==> creating $filesystem $Options on $pacmandb ..."
mkfs.$filesystem $Options $pacmandb > /dev/null 2>&1
       
echo "==> creating temporary mount point $pacman_tmp_mnt_point.."
mkdir $pacman_tmp_mnt_point;
dir $pacman_tmp_mnt_point

echo "==> mounting pacman.db to temporary mount point..."
mount -t $filesystem -o loop $pacmandb $pacman_tmp_mnt_point

echo "==> copying pacman database to temporary mount point..."
cp -a $dbroot. $pacman_tmp_mnt_point

echo "==> unmounting temporary mount point..."
umount $pacman_tmp_mnt_point

echo "==> removing temporary mount point..."
rmdir $pacman_tmp_mnt_point
       
echo "==> moving old /var/lib/pacman to /var/lib/pacman.bak..."
mv $dbroot /var/lib/pacman.bak
       
echo "==> createing new pacman db mount point @ $dbroot..."
mkdir $dbroot

echo "==> Mounting new pacman db..."
mount -t $filesystem -o loop $pacmandb $dbroot

echo "==> md5sum'ing the new database..."
find $dbroot -type f | sort | xargs md5sum >/tmp/pacsums.new

echo "==> checking integrity..."
diff /tmp/pacsums.old /tmp/pacsums.new >/dev/null 2>&1
if [ $? -ne 0 ]; then
        # failed, move the old one back into place
        umount $dbroot
        rm $pacmandb
        mv $dbroot.bak $dbroot
        die_r "integrity check FAILED, reverting to old database"
fi

rm -f /tmp/pacman.lck /tmp/pacsums.old /tmp/pacsums.new

# Replaced by wain: mount is now in /etc/rc.d/pacmandb daemon
#echo "==> Updating /etc/fstab to reflect changes..."
#echo "$pacmandb $dbroot $filesystem loop,defaults 0 0" >> /etc/fstab
 
echo
echo "Finished.  Your pacman database has been caged!.  May the speedy pacman be with you."
echo

exit 0

wain · 2008-08-25 19:00:56

Nice

Just 2 things:
- /tmp/pacman.lck have to be replaced by $(LC_ALL=C pacman -v | grep 'Lock File' | awk '{print $4}') or default /var/lib/pacman/db.lck
- dbroot can be replaced the same way by $(LC_ALL=C pacman -v | grep 'DB Path' | awk '{print $4}')

Do you feel best performance with others options or different filesystem ?

NicolasDiaz · 2008-08-30 16:27:21

Nice ...

I'm trying to use reiserfs, and in my x86 (celeron D 320 (2.4 GHz, 533Mhz FSB ,256Kb cache , 512 Mb Ram and a old 40 Gb PATA disk)) is better reiserfs that ext2 or ext3, but in a x86_64 with 2GB Ram and Intel(R) Core(TM)2 CPU 4300 @ 1.80GHz with 2 cores, the speedup is better with ext2 or ext3.

in the next post, i will post the results,because ssh is not working well in my x86 , I'm far (150Km) from my machine

Fackamato · 2009-04-06 09:11:31

Is anyone using this?

Daenyth · 2009-04-06 12:00:03

pacman-optimize is included in the pacman package

Fackamato · 2009-04-06 13:11:12

Daenyth wrote:

pacman-optimize is included in the pacman package

Uh yeah but RAM is quite much faster than HDD still, isn't it?

moz · 2009-06-06 19:12:14

Hi,
Ive had a strange problem. The filesystem seems to get full after a while, atleast so it seems cos i cant write anymore into it even though theres about 90 megs free. I checked with #tune2fs -l /var/lib/pacman.db and i noticed theres no more free inodes...
If anyone can shed some light on this cos i dont know what to make of it. I had to make a new fs of 300Mb this time and copy stuff over.

tune2fs 1.41.5 (23-Apr-2009)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          7fe88983-7117-4c20-9225-7250b60b0a6e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              38456
Block count:              153600
Reserved block count:     0
Free blocks:              92936
Free inodes:              1
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2024
Inode blocks per group:   253
Filesystem created:       Sat May 23 08:53:54 2009
Last mount time:          Sat Jun  6 19:52:33 2009
Last write time:          Sat Jun  6 20:16:07 2009
Mount count:              3
Maximum mount count:      21
Last checked:             Sat Jun  6 19:06:26 2009
Check interval:           15552000 (6 months)
Next check after:         Thu Dec  3 18:06:26 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Default directory hash:   half_md4
Directory Hash Seed:      0337409d-0db8-4104-b4d0-eb3f39c4f31b

Themaister · 2009-06-07 10:14:30

Wow o.O This is awesome. I don't quite understand what this script does though. Is it something like mounting the database in a tmpfs?

Thikasabrik · 2009-06-07 18:25:42

moz wrote:

Hi,
Ive had a strange problem. The filesystem seems to get full after a while, atleast so it seems cos i cant write anymore into it even though theres about 90 megs free. I checked with #tune2fs -l /var/lib/pacman.db and i noticed theres no more free inodes...
If anyone can shed some light on this cos i dont know what to make of it. I had to make a new fs of 300Mb this time and copy stuff over.

I recommend modifying the script so the fs creation is like this:

mkfs.ext2 -O dir_index -b 1024 -i 1024 -m 0

The key difference being the bytes-per-inode parameter 'i'. My db file is 30mb with these options.

DarkMind · 2009-07-17 00:27:04

very fast, amazing

ruscook · 2009-08-20 22:07:28

I've just decided to enable pacman-cage and yes it is fast! Obviously I'm also concerned that I don't lose data using the loopback mount.

What's I've done is add an rsync backup on successful mounting of the loopback f/s on booting. To do this I modified the rc.local script already posted here (sorry can't remember who at the moment).

Here's the modded script.

#!/bin/bash
#
# /etc/rc.local: Local multi-user startup script.
#
. /etc/rc.conf
. /etc/rc.d/functions


# Check and mount pacman database
stat_busy "Mounting the pacman database"
/sbin/e2fsck -p /var/lib/pacman-X.db
if [ $? -gt 1 ]; then
   stat_fail
   echo
   echo "WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED"
   echo
else
   mount /var/lib/pacman
   if [ $? -gt 0 ]; then
      stat_fail
      echo
      echo "WARNING: MOUNTING PACMAN DATABASE FAILED"
      echo
   else
      stat_done
    stat_busy "Backing up the pacman database"    
    rsync -Caxq --delete-after --exclude "lost+found/" /var/lib/pacman/  /var/lib/pacman-backup/
      stat_done
   fi
fi

As it's rsync and only changed files, it's pretty quick so I'm not worried about speed. I can also cron this if I wanted to ensure backups between boots.

Can any tell me if there's any hidden gotcha's (issues) with this idea or is it safe to do the rsync on each boot?
Thanks
Russ

wain · 2009-08-21 07:04:03

ruscook wrote:

Can any tell me if there's any hidden gotcha's (issues) with this idea or is it safe to do the rsync on each boot?

I don't know but you just have to backup /var/lib/pacman/local/. It's faster.

I use pacman-cage for many years without any problem.

ruscook · 2009-08-24 11:13:07

wain wrote:

ruscook wrote:
Can any tell me if there's any hidden gotcha's (issues) with this idea or is it safe to do the rsync on each boot?
I don't know but you just have to backup /var/lib/pacman/local/. It's faster.
I use pacman-cage for many years without any problem.

Thanks Wain. Why only local/ not sync/ as well?

Russ

wain · 2009-08-24 11:25:48

ruscook wrote:

Thanks Wain. Why only local/ not sync/ as well?

Because "sync" directory is created by pacman -Sy and don't contain any personnal info

ruscook · 2009-08-27 08:29:53

Ah even better! Thanks, will adjust my script.
Russ

Arch Linux

#51 2007-01-18 08:41:46

Re: Pacman Optimization - Part 2

#52 2007-01-18 21:11:48

Re: Pacman Optimization - Part 2

#53 2007-01-19 17:30:00

Re: Pacman Optimization - Part 2

#54 2007-01-22 17:22:08

Re: Pacman Optimization - Part 2

#55 2007-11-26 23:03:52

Re: Pacman Optimization - Part 2

#56 2007-11-27 02:22:11

Re: Pacman Optimization - Part 2

#57 2008-05-09 06:51:25

Re: Pacman Optimization - Part 2

#58 2008-06-14 16:00:39

Re: Pacman Optimization - Part 2

#59 2008-06-14 19:29:22

Re: Pacman Optimization - Part 2

#60 2008-07-03 07:41:12

Re: Pacman Optimization - Part 2

#61 2008-08-25 17:29:06

Re: Pacman Optimization - Part 2

#62 2008-08-25 19:00:56

Re: Pacman Optimization - Part 2

#63 2008-08-30 16:27:21

Re: Pacman Optimization - Part 2

#64 2009-04-06 09:11:31

Re: Pacman Optimization - Part 2

#65 2009-04-06 12:00:03

Re: Pacman Optimization - Part 2

#66 2009-04-06 13:11:12

Re: Pacman Optimization - Part 2

#67 2009-06-06 19:12:14

Re: Pacman Optimization - Part 2

#68 2009-06-07 10:14:30

Re: Pacman Optimization - Part 2

#69 2009-06-07 18:25:42

Re: Pacman Optimization - Part 2

#70 2009-07-17 00:27:04

Re: Pacman Optimization - Part 2

#71 2009-08-20 22:07:28

Re: Pacman Optimization - Part 2

#72 2009-08-21 07:04:03

Re: Pacman Optimization - Part 2

#73 2009-08-24 11:13:07

Re: Pacman Optimization - Part 2

#74 2009-08-24 11:25:48

Re: Pacman Optimization - Part 2

#75 2009-08-27 08:29:53

Re: Pacman Optimization - Part 2

Board footer