even more faster pacman searches ??

solstice · 2006-12-16 11:13:37

UPDATED: version 0.2.1
rewritten synchronize function. it is more robust now.
UPDATED: version 0.2
fixes a SERIOUS bug:
directories of the db were not deleted when the db is synchronize back in case some packages had been removed. (i am a newbie with the use of rsync) hopefully that bug will not hit many people because the number of user of the script might be very low.
If you're the user hit by the bug: for upgraded package simply delete the duplicate directory in /var/lib/pacman/local
for removed package, you have to track which one have been removed, and remove the directory in /var/lib/pacman/local.
I will update my python script to check for that (see below in the thread)
UPDATED: version 0.1.2
- add a synchronize option (it simply synchronize back the db to disk but keep pacman-tmpfs running)
-basic refactoring (bad ?)
UPDATED: version 0.1.1
- add a restart option
- refactoring: use a cleanup fonction

hi,
i was looking at those pacman-cache and pacman-drive and i was wondering why do they use ext2 fs in a loopback ? and i already have ext3 for my / so if i use one of them will it give me any speed improvement ?

So it thought why not use tmpfs. all in ram it must be super fast ! :shock:

so i made pacman-tmpfs (i have not thought very long to find a name )

it is a daemon script to be placed in /etc/rc.d and to be activated in DAEMONS of /etc/rc.conf (or manually run first may be)

it copies the pacman db in a tmpfs filesystem and move it over the pacman db. DO NOT PANIC. it is not overwritten. and then when you stop the daemon it sync back the change to the original pacman db
$ time pacman -Ss pacman
...
real 0m1.229s
user 0m0.596s
sys 0m0.568s

you could still use pacman-optimize but pacman-tmpfs must be stopped so that it does something usefull. Using pacman-optimize will give you better start-up time for pacman-tmpfs.

do not use pacman-tmpfs while running pacman-drive or pacman-cache.

I run pacman-tmpfs for only 1 day. so it lacks some big testing . be carefull !
it need testers. BACKUP your pacman db PLEASE !

i have not tested the catching error code i have written in it.

what do you think of it ? please tell me ! :-)

#!/bin/sh
#
# /etc/rc.d/pacman-tmpfs                             Version 0.2.1    2006-12-16
#
# under the WTFPL. see http://sam.zoy.org/wtfpl/
#
#             DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
#                     Version 2, December 2004 
# 
#  Copyright (C) 2006 solsTiCe d'Hiver
#                           <solstice>
#  Everyone is permitted to copy and distribute verbatim or modified 
#  copies of this license document, and changing it is allowed as long 
#  as the name is changed. 
# 
#             DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
#    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 
# 
#   0. You just DO WHAT THE FUCK YOU WANT TO.


umask 022

. /etc/rc.conf
. /etc/rc.d/functions

pacman_db=/var/lib/pacman
#db_size=256M

# look for a command to synchronize directory
if which dirsync >/dev/null 2>&1 ;then
    sync_cmd="dirsync -q"
# elif which mirdir >/dev/null 2>&1 ;then
#     sync_cmd="mirdir -e none"
elif which rsync >/dev/null 2>&1 ; then
    sync_cmd="rsync -a --delete"
fi

if [ -z "$sync_cmd" ] ; then
    echo "Cannot be run without rsync or dirsync or mirdir"
    exit 1
fi

if [ -f /tmp/pacman.lck ] ;then
    echo "Cannot be run while pacman is running"
    exit 1
fi

mount_move() {
    local ret
    # fix a bug in mount ???
    mount --move "$1" "$2"
    ret=$?
    umount "$1" 2>/dev/null || true
    return $ret
}

_backup() {
    # copy db in /root as backup and give up
    backup_dir=/root/pacman-tmpfs-backup-`date "+%Y-%m-%d_%H:%M:%S"`
    mkdir -p "$backup_dir"
    cp -a $1/* "$backup_dir"
    umount $1
    logger -s -t '[pacman-tmpfs]' -p local0.crit "Cannot synchronize back the database"
    logger -s -t '[pacman-tmpfs]' -p local0.crit "You need to manually restore the content of $backup_dir into $pacman_db"
    logger -s -t '[pacman-tmpfs]' -p local0.crit "You need to delete /var/run/daemon/pacman-tmpfs once that is done."
}

start() {
    # lock pacman db
    touch /tmp/pacman.lck

    tmp_db=`mktemp -d -t tmp_db.XXXXXX`

    # create the tmp fs 
    mount -t tmpfs -o size=$db_size,mode=0755 none $tmp_db
    # copy the db into the tmpfs
    cp -a $pacman_db/* $tmp_db
    # replace the pacman db with tmpfs one
    mount_move $tmp_db $pacman_db

    rmdir $tmp_db

    # unlock pacman db
    rm -f /tmp/pacman.lck
}

stop() {
    #lock pacman db
    touch /tmp/pacman.lck

    tmp_db=`mktemp -d -t tmp_db.XXXXXX`

    # move the tmpfs db so that we can copy the changes back to home
    mount_move $pacman_db $tmp_db
    $sync_cmd $tmp_db/ $pacman_db
    if [ $? -gt 0 ] ;then
        _backup $tmp_db
        stat_fail
        return 1
    fi

    umount $tmp_db
    rmdir $tmp_db

    # unlock pacman db
    rm -f /tmp/pacman.lck
}

synchronize() {
    # we create another tmpfs, copy into it the db (tmpfs -> tmpfs)
    #lock pacman db
    touch /tmp/pacman.lck

    tmp_db=`mktemp -d -t tmp_db.XXXXXX`

    # create a new tmp fs 
    mount -t tmpfs -o size=$db_size,mode=0755 none $tmp_db
    # copy the db(old tmpfs) into the new tmpfs
    cp -a $pacman_db/* $tmp_db
    # get rid off the old tmpfs
    umount $pacman_db

    # we can copy the changes back to home
    $sync_cmd $tmp_db/ $pacman_db
    if [ $? -gt 0 ] ;then
        _backup $tmp_db
        stat_fail
        return 1
    fi

    # now move back inplace the new tmpfs db
    mount_move $tmp_db $pacman_db
    rmdir $tmp_db

    rm -f /tmp/pacman.lck
}

case "$1" in
    start)
        if ck_daemon pacman-tmpfs ;then
            stat_busy "Starting pacman-tmpfs ..."
            start
            if [ $? -gt 0 ] ;then
                stat_fail
            else
                add_daemon pacman-tmpfs
                stat_done
            fi
        else
            stat_fail
        fi
        ;;
    stop)
        if ! ck_daemon pacman-tmpfs ;then
            stat_busy "Stopping pacman-tmpfs. Please wait ..."
            stop
            if [ $? -gt 0 ] ;then
                stat_fail
            else
                rm_daemon pacman-tmpfs
                stat_done
            fi
        else
            stat_fail
        fi
        ;;
    restart)
        if ! ck_daemon pacman-tmpfs ;then
            stat_busy "Stopping pacman-tmpfs. Please wait ..."
            stop
            if [ $? -gt 0 ] ;then
                stat_fail
            else
                rm_daemon pacman-tmpfs
                stat_done
                sleep 1
                stat_busy "Starting pacman-tmpfs ..."
                start
                if [ $? -gt 0 ] ;then
                    stat_fail
                else
                    add_daemon pacman-tmpfs
                    stat_done
                fi
            fi
        else
            stat_fail
        fi
        ;;
    synchronize)
        if ! ck_daemon pacman-tmpfs ;then
            stat_busy "Synchronizing ..."
            synchronize
            if [ $? -gt 0 ] ;then
                stat_fail
            else
                stat_done
            fi
        else
            stat_fail
        fi
        ;;
    *)
        echo "Usage: $0 {start|stop|restart|synchronize}"
        ;;
esac
exit 0
# vim:ts=4:sw=4

iphitus · 2006-12-16 11:29:59

should just remind people that, if while using this setup, you lose power, you lose your up to date pacman db.

solstice · 2006-12-16 11:51:13

you loose your up to date pacman db

yes. that's a big problem of that script. but you still have on disk the pacman db you had when you launched the script.

it could be restarted in a cron job to sync back the db. (?)
but that's not a perfect solution

it's fast but you could loose your pacman db even faster ;-)

EDIT:
but with the help of /var/log/pacman.log, you still can reinstall the package that are missing in your /var/lib/pacman/local if your pc loose power and loose too the pacman db on tmpfs.

i might create a script to ease that recover process.

noriko · 2006-12-16 18:19:59

y not sync after each update?
it only takes a second ...
run it in the background..
just a thought... otherwise kool script, i look forward to trying it whne i get bak on my pc 8)

xerverius · 2006-12-16 19:34:30

you can also create something like this in your .bash_profile:

alias pacman='pacmansync'

pacmansync ()
{
  pacman $@
  
  #sync code here
}

This way the data is always synced after you run pacman.

Fackamato · 2006-12-17 13:38:13

Nice script

It's not much faster on my system, a couple of milliseconds, but it's faster

noriko · 2006-12-17 16:11:09

hmmm

so i saw the pacman search speedu train and decided to jump on it an hour ago...lol
[heavily edited]
now i have a php script that search the entire official db`s .. 3757 pkgs in milliseconds..
it also supports php compatible regex ...
searching for "(.*)" aka everthing returns in 3 seconds... most of that time is taken up by the echo/print statement...

i will post my result to a new thread latre when i get on my pc ...
i won't say how i cheated to do that yet lol don't want anyone utilising this...
then writes something that that totally pwns mine before i go downstairs lol...
all i will say is that it doesn't even use a ramdisk x)

solstice · 2006-12-18 18:40:57

that script checks for any differences between pacman.log and /var/lib/pacman/local
it prints package that must be installed according to pacman.log but not found in pacman's local db

#!/usr/bin/env python

# pacman.py                                        Version 0.2      2006-12-16
#
# under the WTFPL. see http://sam.zoy.org/wtfpl/
#
#             DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
#                     Version 2, December 2004 
# 
#  Copyright (C) 2006 solsTiCe d'Hiver
#                           <solstice>
#  Everyone is permitted to copy and distribute verbatim or modified 
#  copies of this license document, and changing it is allowed as long 
#  as the name is changed. 
# 
#             DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
#    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 
# 
#   0. You just DO WHAT THE FUCK YOU WANT TO.

import os
import os.path
from sys import exit,argv
import re
import urllib

class Pkg:
    def __init__(self,n,v,r):
        self.name = n
        self.ver = v
        self.rel = r

    def __str__(self):
        return '%s %s-%s' % (self.name,self.ver,self.rel)

    def __cmp__(self,other):
        return cmp(self.name,other.name)

def create_pkg(s):
    words = s.split('-')
    name = words[0:-2]
    words[0:-2] = ['-'.join(name)]
    return Pkg(words[0],words[1],words[2])

def init_db():
    global db
    db = {}
    repositories = os.listdir(dbdir)
    for r in repositories:
        repodir = os.sep.join([dbdir,r])
        if not os.path.isdir(repodir):
            continue
        tmp = os.listdir(repodir)
        try:
            tmp.remove('.lastupdate')
        except ValueError:
            pass
        tmp.sort()
        db[r] = tmp

def from_repo(s):
    # find pkg from repo s that are installed locally
    tmp = []
    try:
        for p in db['local']:
            if p in db[s]:
                tmp.append(create_pkg(p))
    except KeyError:
        pass
    tmp.sort()
    return tmp

def localpkg():
    # find pkg installed ( i.e. in local) but not found in any repo
    keys = db.keys()
    keys.remove('local')
    tmp = []
    for p in db['local']:
        found = False
        for r in keys:
            if p in db[r]:
                found = True
                break
        if not found:
            tmp.append(create_pkg(p))
    tmp.sort()
    return tmp

def parselog(initlog = None,verbose=True):
    installed = re.compile(r'.*installed (.*) ((.*))')
    removed = re.compile(r'.*removed (.*) ((.*))')
    upgraded = re.compile(r'.*upgraded (.*) ((.*) -> (.*))')

    result = []
    if initlog is not None:
        for line in initlog:
            if not line.startswith('base/'):
                continue
            line = line.replace('base/','')
            line = line.replace('.pkg.tar.gz','')
            result.append(create_pkg(line.rstrip()))

    try:
        log = open('/var/log/pacman.log','r')
        for line in log:
            m = installed.match(line)
            if m:
                ll = m.groups()
                result.append(create_pkg('-'.join([ll[0],ll[1]])))
            m = removed.match(line)
            if m:
                ll = m.groups()
                pkg = create_pkg('-'.join([ll[0],ll[1]]))
                try:
                    result.remove(pkg)
                except ValueError:
                    if verbose:
                        print 'Warning: %s not installed but removed' % pkg
                    pass
            m = upgraded.match(line)
            if m:
                ll = m.groups()
                pkg = create_pkg('-'.join([ll[0],ll[1]]))
                try:
                    result.remove(pkg)
                except ValueError:
                    if verbose:
                        print 'Warning: %s not installed but upgraded' % pkg
                    pass
                result.append(create_pkg('-'.join([ll[0],ll[2]])))

        log.close()
        return result
    except IOError,e:
        print '%s: %s' % (e.filename,e.strerror)
        exit(1)

def difference(u,v):
    diff = []
    for i in u:
        if i not in v:
            diff.append(i)
    return diff

def duplicate(l):
    result = []
    i = 0
    while i <len> 1 and argv[1] == '-v':
        print ':: Packages that are installed according to pacman.log'
        print 'n'.join('%s' % i for i in pll)
        exit(0)
    dblocal = [create_pkg(i) for i in db['local']]
    diff = difference(pll,dblocal)
    if diff:
        print ''':: Installed according to pacman.log but not found in /var/lib/pacman/local'''
        print 'n'.join('%s' % i for i in diff)
    diff = difference(dblocal,pll)
    if diff:
        print ''':: Found in /var/lib/pacman/local but not installed according to pacman.log'''
        print 'n'.join('%s' % i for i in diff)
    dup = duplicate(dblocal)
    if dup:
        print ":: Duplicates found in pacman's db"
        print 'n'.join('%s' % i for i in dup)

noriko · 2006-12-18 19:16:57

/port/bin/pmd: line 22: class: command not found
/port/bin/pmd: line 23: syntax error near unexpected token `('
/port/bin/pmd: line 23: ` def __init__(self,n,v,r):'

that's prolly something to do with my system, bu ti dunno so i post it anyway... any ideas on what may have caused this?

Cerebral · 2006-12-18 19:28:18

I'd suggest wikifying these - the forum sometimes screws up pasted stuff.

solstice · 2006-12-18 20:21:39

i can't find any error. how do you run it ?

I must have mentionned that the latest script is a python script. to be run with python.

noriko · 2006-12-18 20:34:42

i run it with python | py pmd
bash | sh pmd
and as is | pmd

i just copied and pasted it, maybe there is a formatting error somewhere,
have u got it like, on a webserver somewhere? where i can download the file directly?

edit .. nevr mind

it works now, seemed to have been some sorta formatting eror as suspected, it had a bunch of non-roman/english characters in it, when opened in nano .. same as there usually is when u open windows text files created with something liek wordpad...

noriko · 2006-12-18 20:43:10

might i just add, this script has saved em a lot of future pain, since i accidentally deleted the pdb/local .. i meant to get rid of pbd/localhost ..
guess someone will be choosing better names in teh future lool..
i think i ws only missing 20 or so pkgs anyway, i managed to rebuild most of teh db by installing pkg groups likes, gnome...

Arch Linux

#1 2006-12-16 11:13:37

even more faster pacman searches ??

#2 2006-12-16 11:29:59

Re: even more faster pacman searches ??

#3 2006-12-16 11:51:13

Re: even more faster pacman searches ??

#4 2006-12-16 18:19:59

Re: even more faster pacman searches ??

#5 2006-12-16 19:34:30

Re: even more faster pacman searches ??

#6 2006-12-17 13:38:13

Re: even more faster pacman searches ??

#7 2006-12-17 16:11:09

Re: even more faster pacman searches ??

#8 2006-12-18 18:40:57

Re: even more faster pacman searches ??

#9 2006-12-18 19:16:57

Re: even more faster pacman searches ??

#10 2006-12-18 19:28:18

Re: even more faster pacman searches ??

#11 2006-12-18 20:21:39

Re: even more faster pacman searches ??

#12 2006-12-18 20:34:42

Re: even more faster pacman searches ??

#13 2006-12-18 20:43:10

Re: even more faster pacman searches ??

Board footer