Pacman Optimization - Part 2

slyson · 2009-09-19 10:26:13

I have the standard kernel 2.6.30-ARCH and ext4 filesystem.

sudo pacman-cage:

==> md5sum'ing the old database...
==> creating pacman.db loopback file...
==> creating ext2 -O dir_index -b 1024 on /var/lib/pacman.db...
==> creating temporary mount point /mnt/tmp-pacman..
==> mounting pacman.db to temporary mount point...
Error: could not open any loop device
==> copying pacman database to temporary mount point...
==> unmounting temporary mount point...
umount: /mnt/tmp-pacman: not mounted
==> removing temporary mount point...
rmdir: failed to remove `/mnt/tmp-pacman': Directory not empty
==> moving old /var/lib/pacman to /var/lib/pacman.bak...
==> createing new pacman db mount point @ /var/lib/pacman...
==> Mounting new pacman db...
Error: could not open any loop device
==> md5sum'ing the new database...
==> checking integrity...
umount: /var/lib/pacman: not mounted
pacman-cage: integrity check FAILED, reverting to old database

Why can't it open any loopback device?

Last edited by slyson (2009-09-19 10:29:04)

Zariel · 2009-09-19 10:52:43

did you modprobe loop ?

slyson · 2009-09-19 18:34:50

Zariel wrote:

did you modprobe loop ?

sudo modprobe loop

FATAL: Could not load /lib/modules/2.6.30-ARCH/modules.dep: No such file or directory

Ranguvar · 2009-09-19 20:56:49

Try depmod `uname -r` and then the modprobe.

Burning_aces · 2010-01-04 17:09:35

ok so i just updated to the latest kernel and now the pacmandb doesnt mount so i tried running the script from the first page here, then the pacman-uncage neither worked. then rm -r'd the tmp/pacman thing
and now i cant remove or install anything
how do i go about fixing this?

Dr4go · 2010-01-27 05:22:17

Thanks for the great pacman-cage script!

Because I'm using pacman-cage also inside of an arch32 chroot, I've modified the /etc/rc.d/arch32 daemon script also to reflect the pacman-cage mounting (normally done in /etc/rc.d/pacmandb) as follows:

#!/bin/bash

. /etc/rc.conf
. /etc/rc.d/functions

case $1 in
    start)
        stat_busy "Starting Arch32 chroot"
        mount --bind /proc /opt/arch32/proc
        mount --bind /proc/bus/usb /opt/arch32/proc/bus/usb
        mount --bind /dev /opt/arch32/dev
        mount --bind /dev/pts /opt/arch32/dev/pts
        mount --bind /dev/shm /opt/arch32/dev/shm
        mount --bind /sys /opt/arch32/sys
        mount --bind /tmp /opt/arch32/tmp
        mount --bind /home /opt/arch32/home
        /sbin/e2fsck -p /opt/arch32/var/lib/pacman.db >/dev/null
        if [ $? -gt 1 ]; then
                stat_fail
                echo 
                echo "WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED" 
                echo 
        else
                mount -o loop -t ext2 /opt/arch32/var/lib/pacman.db /opt/arch32/var/lib/pacman
                if [ $? -gt 0 ]; then
                        stat_fail
                        echo 
                        echo "WARNING: MOUNTING PACMAN DATABASE FAILED" 
                        echo 
                else
                        add_daemon arch32
                        stat_done
                fi
        fi
        ;;
    stop)
        stat_busy "Stopping Arch32 chroot"
        umount /opt/arch32/proc/bus/usb
        umount /opt/arch32/proc
        umount /opt/arch32/dev/pts
        umount /opt/arch32/dev/shm
        umount /opt/arch32/dev
        umount /opt/arch32/sys
        umount /opt/arch32/tmp
        umount /opt/arch32/home
        umount /opt/arch32/var/lib/pacman >/dev/null 2>&1
        if [ $? -ne 0 ]; then
                stat_fail
                echo 
                echo "WARNING: UNMOUNTING PACMAN DATABASE FAILED" 
                echo 
        else
                rm_daemon arch32
                stat_done
        fi
        ;;
    restart)
        $0 stop
        sleep 1
        $0 start
        ;;
    *)
        echo "usage: $0 {start|stop|restart}"
esac
exit 0

Please note: You have to perform a pacman-cage inside of your arch32 chroot before using the modified script!

Last edited by Dr4go (2010-01-27 05:59:42)

DonVla · 2010-02-11 15:01:10

Hello, there are some minor flaws.
First, if there is already a /var/lib/pacman.bak directory, pacman-cage moves the old pacman dir into the pacman.bak dir. I think this is not intended. Line 105/6 in the pacman-cage script I would change to

mv /var/lib/pacman /var/lib/pacman.newbak
[[ -d /var/lib/pacman.bak ]] && rm -fr /var/lib/pacman.bak
mv /var/lib/pacman.newbak /var/lib/pacman.bak

Then it would be nice if the pacman-[un]cage scripts will write a notice to the pacman.log file that the db was [un]caged.
And last for the totally paranoid I think a cronjob backing up the caged pacman directory (or at least pacman/local) may be helpful.

Vlad

PS: I've uploaded a source tarball with some minor PKGBUILD chnages and a patch which add the functionality mentioned above, ie
* remove the pacman.bak dir before creating a new one
* add pacman-cage notes to pacman.log
* added a cron script which backs up pacman's local db (default backup dir is /var/backups/pacman_local_database).
You can dl the tarbal here: www.nakamura-gebiet.de/pacman-cage-2.9.8.2-5.src.tar.gz

Last edited by DonVla (2010-02-13 15:46:58)

algorythm · 2010-02-27 09:27:31

I tried how fast would an ext4 cage be but it turned out that because ext2 takes much less space - ext4 cage was actually slower (I mean I didn't even have journal in it).

It might also effect that it is database what we are talking about...

Last edited by algorythm (2010-02-27 19:25:47)

fthieme · 2010-03-21 19:32:08

You can gain some speed when you mount the loopback with noatime, because the access time of each file is tracked otherwise...

noalwin · 2010-04-07 21:27:12

algorythm wrote:

I tried how fast would an ext4 cage be but it turned out that because ext2 takes much less space - ext4 cage was actually slower (I mean I didn't even have journal in it).
It might also effect that it is database what we are talking about...

what about an reiserfs cage?
It should be faster

SanskritFritz · 2010-04-07 21:45:40

I also thought of this, shouldnt be too hard to try? Reiserfs is reportedly fast on many small files.

SanskritFritz · 2010-04-07 23:18:04

fthieme wrote:

You can gain some speed when you mount the loopback with noatime, because the access time of each file is tracked otherwise...

Agreed, noatime should be included:

mount -o loop,noatime -t ext2 /var/lib/pacman.db /var/lib/pacman

I wasnt able to measure any speed gain though, so this could be purely theoretical...

SanskritFritz · 2010-04-27 12:25:09

A little patch, eliminating the spaces around the date in the logfile (looks not uniform) and corrects a typo:

diff Builds/bauerbill/aur/pacman-cage/src/pacmandb.rc /etc/rc.d/pacmandb

13c13
<     echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> Mounting the pacmans database" >> ${pacmanlog}
---
>     echo "[$(date "+%Y-%m-%d %H:%M")] Pacman-cage [rc] >> Mounting the pacman database" >> ${pacmanlog}
21c21
<         echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> WARNING: Pacman database filesystem check failed, not mounted!" >> ${pacmanlog}
---
>         echo "[$(date "+%Y-%m-%d %H:%M")] Pacman-cage [rc] >> WARNING: Pacman database filesystem check failed, not mounted!" >> ${pacmanlog}
23c23
<         mount -o loop -t ext2 /var/lib/pacman.db /var/lib/pacman
---
>         mount -o loop,noatime -t ext2 /var/lib/pacman.db /var/lib/pacman
30c30
<             echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> WARNING: Mounting of the pacman database failed!" >> ${pacmanlog}
---
>             echo "[$(date "+%Y-%m-%d %H:%M")] Pacman-cage [rc] >> WARNING: Mounting of the pacman database failed!" >> ${pacmanlog}
35c35
<             echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> Pacman database successfully mounted." >> ${pacmanlog}
---
>             echo "[$(date "+%Y-%m-%d %H:%M")] Pacman-cage [rc] >> Pacman database successfully mounted." >> ${pacmanlog}
45c45
<         echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> WARNING: Umounting of the pacman database failed!" >> ${pacmanlog}
---
>         echo "[$(date "+%Y-%m-%d %H:%M")] Pacman-cage [rc] >> WARNING: Umounting of the pacman database failed!" >> ${pacmanlog}
51c51
<         echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> Pacman database successfully unmounted." >> ${pacmanlog}
---
>         echo "[$(date "+%Y-%m-%d %H:%M")] Pacman-cage [rc] >> Pacman database successfully unmounted." >> ${pacmanlog}

fthieme · 2010-04-27 16:32:48

noalwin wrote:

what about an reiserfs cage?
It should be faster

After switching to reiserfs away from XFS I see that pacman-cage is not a real improvement, if you have a filesystem that is fast with small files. It was a huge improvement while using XFS, tough...

algorythm · 2010-06-19 16:47:42

Huge improvement when compared to ext4 and/or reiserfs? The people here would disagree a bit: http://bbs.archlinux.org/viewtopic.php?id=80065..

Also @noalwin, I'd rather test with reiser4 than reiser3 when I just have time to rebuild my kernel with a reiser4 patch. But It could even better peformance to actually just use the whole /var as reiser3/4 or whatever fs'd be good with a lot of small files.

SanskritFritz · 2010-06-19 18:23:21

It is maybe worth mentioning here:
http://xyne.archlinux.ca/projects/rebase
I use now both rebase with pacman-cage, blazing fast.

SanskritFritz · 2010-07-26 22:55:57

Now that pacman 3.4 is out, maybe pacman-cage isnt needed anymore? My tests show not much of a difference anymore.

algorythm · 2010-07-27 18:31:58

Listen to his crazy face.

But did you check the difference with:

time pacman -Ss pacman

Last edited by algorythm (2010-07-27 18:49:46)

SanskritFritz · 2010-07-27 19:58:39

No, as it is difficult to achieve the same result twice. I just did a pacman-uncage some days ago, and pacman feels the same now.

algorythm · 2010-07-27 20:03:51

You don't need to achieve the same result because 1) without the cage pacman is (at least was) MUCH slower and 2) you can write down like three results with both to have a better idea about the speed with each.

There was a little (like 1 second) delay with me when doing "pacman -Syu" before anything started to happen with ArchBang with non-caged pacman.

So I'll just keep using it.

Last edited by algorythm (2010-07-27 20:06:26)

xrchz · 2010-09-22 23:53:48

Apart from the speed (which is a general benefit), are there any particular benefits to using pacman-cage on an SSD?
In particular, would it reduce the number of disk writes?

I am thinking of using both rebase and pacman-cage on my new SSD when I get it. Same question for rebase, but since rebase is all about reducing disk IO the answer is probably "yes" there...

Allan · 2010-09-23 00:10:00

pacman already does what rebase does (without the output). pacman-cage will not reduce disk writes.

xrchz · 2010-09-23 00:29:28

Ah thanks Allan. I'll reupdate the relevant wiki pages.

mr.schyte · 2011-03-05 11:33:21

I've modified the script to create a RAID1 ramdisk for even more speed

#!/bin/bash

. /etc/rc.conf
. /etc/rc.d/functions

function initialise() {
    /bin/cp /var/lib/pacman.db /dev/shm >/dev/null 2>&1

    /sbin/losetup /dev/loop0 /dev/shm/pacman.db >/dev/null 2>&1
    /sbin/losetup /dev/loop1 /var/lib/pacman.db >/dev/null 2>&1

    /sbin/mdadm --assemble --scan >/dev/null 2>&1
}

function teardown() {
    stat_busy "Unmounting the pacman database"
    umount /var/lib/pacman >/dev/null 2>&1

    /sbin/mdadm --stop /dev/md/md0 >/dev/null 2>&1
    
    /sbin/losetup -d /dev/loop0 >/dev/null 2>&1
    /sbin/losetup -d /dev/loop1 >/dev/null 2>&1

    /bin/rm /dev/shm/pacman.db >/dev/null 2>&1
}

pacmanlog=$(pacman -v | grep "Log File" | cut -d ":" -f2)

case "$1" in
    start)
    # Check and mount pacman database 
    stat_busy "Mounting the pacman database"
    # write to pacman.log
    initialise

    echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> Mounting the pacmans database" >> ${pacmanlog}
    /sbin/e2fsck -p /dev/md/md0 >/dev/null

    if [ $? -gt 1 ]; then
        stat_fail
        echo 
        echo "WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED" 
        echo 
        # write to pacman.log
        echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> WARNING: Pacman database filesystem check failed, not mounted!" >> ${pacmanlog}
    else
        mount -t ext2 /dev/md/md0 /var/lib/pacman
        if [ $? -gt 0 ]; then
            stat_fail
            echo 
            echo "WARNING: MOUNTING PACMAN DATABASE FAILED" 
            echo 
            # write to pacman.log
            echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> WARNING: Mounting of the pacman database failed!" >> ${pacmanlog}
            teardown
        else
            stat_done
            add_daemon pacmandb
            # write to pacman.log
            echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> Pacman database successfully mounted." >> ${pacmanlog}
        fi
    fi
    ;;

    stop)
    teardown
    if [ $? -ne 0 ]; then
        # write to pacman.log
        echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> WARNING: Umounting of the pacman database failed!" >> ${pacmanlog}
        stat_fail
    else
        stat_done
        rm_daemon pacmandb
        # write to pacman.log
        echo "[ $(date "+%Y-%m-%d %H:%M ")] Pacman-cage [rc] >> Pacman database successfully unmounted." >> ${pacmanlog}
    fi
    ;;

    restart)
    $0 stop
    sleep 2
    $0 start
    ;;

    *)
    echo "usage: $0 {start|stop|restart}"  
    ;;
esac

SanskritFritz · 2011-03-23 12:41:28

Will the advent of pacman 3.5 make this script obsolete? I believe so, opinions please?

Arch Linux

#76 2009-09-19 10:26:13

Re: Pacman Optimization - Part 2

#77 2009-09-19 10:52:43

Re: Pacman Optimization - Part 2

#78 2009-09-19 18:34:50

Re: Pacman Optimization - Part 2

#79 2009-09-19 20:56:49

Re: Pacman Optimization - Part 2

#80 2010-01-04 17:09:35

Re: Pacman Optimization - Part 2

#81 2010-01-27 05:22:17

Re: Pacman Optimization - Part 2

#82 2010-02-11 15:01:10

Re: Pacman Optimization - Part 2

#83 2010-02-27 09:27:31

Re: Pacman Optimization - Part 2

#84 2010-03-21 19:32:08

Re: Pacman Optimization - Part 2

#85 2010-04-07 21:27:12

Re: Pacman Optimization - Part 2

#86 2010-04-07 21:45:40

Re: Pacman Optimization - Part 2

#87 2010-04-07 23:18:04

Re: Pacman Optimization - Part 2

#88 2010-04-27 12:25:09

Re: Pacman Optimization - Part 2

#89 2010-04-27 16:32:48

Re: Pacman Optimization - Part 2

#90 2010-06-19 16:47:42

Re: Pacman Optimization - Part 2

#91 2010-06-19 18:23:21

Re: Pacman Optimization - Part 2

#92 2010-07-26 22:55:57

Re: Pacman Optimization - Part 2

#93 2010-07-27 18:31:58

Re: Pacman Optimization - Part 2

#94 2010-07-27 19:58:39

Re: Pacman Optimization - Part 2

#95 2010-07-27 20:03:51

Re: Pacman Optimization - Part 2

#96 2010-09-22 23:53:48

Re: Pacman Optimization - Part 2

#97 2010-09-23 00:10:00

Re: Pacman Optimization - Part 2

#98 2010-09-23 00:29:28

Re: Pacman Optimization - Part 2

#99 2011-03-05 11:33:21

Re: Pacman Optimization - Part 2

#100 2011-03-23 12:41:28

Re: Pacman Optimization - Part 2

Board footer