You are not logged in.

#1 2009-05-15 23:33:55

BoySka
Member
Registered: 2008-05-09
Posts: 25

pacatatime - Upgrade your sistem one package at a time

Hi all, I want to show you a small piece of code I wrote, I hope you like it.
http://aur.archlinux.org/packages.php?ID=26416
I'll explain it's usage

# pacatatime a long list of  packages that will probably fill up my disk

will "split" it in:

#pacman  -S a long list
#pacman -S of packages that
#pacman -S will probably fill
#pacman -S up my disk
# pacatatime

will UPGRADE your system few packages at a time, like above

This is useful especially if you have little disk space, like I have on my EeePc.
Upgrading my system was going to be a problem, and I often found myself doing "pacatatime" by hand. Not anymore smile

Any suggestion is welcome

Last edited by BoySka (2009-05-15 23:36:15)

Offline

#2 2009-05-16 17:32:22

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

this doesn't appear to work, it leaves some packages un-upgraded
also, it'd prolly be more reliable to upgrade the packages more directly than trying to parse the pacman output
maybe something like this, (i know it's a little destructive installing without dependencies)

#!/bin/sh

for url in $(pacman -Syup | grep '://'); do
    pacman -Ud "$url" || echo 'Upgrade Failed ... Aborting' && exit 1
    yes | pacman -Scc > /dev/null
done

you can apply this to your python script, i.e get and filter the urls, then upgrade them directly pacman -U
i don't know the usefulness of -t as you can only upgrade one package at a time anyway

Last edited by kumyco (2009-05-16 17:34:20)

Offline

#3 2009-05-16 17:49:03

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Re: pacatatime - Upgrade your sistem one package at a time

Moving to community contributions smile.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#4 2009-05-17 21:36:52

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

kumyco wrote:

this doesn't appear to work, it leaves some packages un-upgraded

can you give me more info? It'll be useful for debugging wink (probably it's regexp errors)

for url in $(pacman -Syup | grep '://'); do

I like this idea, I think I'll use it because it's simple to parse (less-errors!) and give you dependencies in a simple way.

    pacman -Ud "$url" || echo 'Upgrade Failed ... Aborting' && exit 1

I don't like this very much, since it could be dangerous for your system. I'll put it as an option, but won't be the default.

i don't know the usefulness of -t as you can only upgrade one package at a time anyway

-t specifies the number of packages to install at one time. BTW, the default is not 1 but 3.

I'll work on your ideas, and more smile

Last edited by BoySka (2009-05-17 22:02:38)

Offline

#5 2009-05-17 22:53:55

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

i know, my point was that when you do pacman -S foo, it will resolve the dependencies(all of them) and install everything which sorta goes against what you were trying to do.

the issues with not installing some packages came from cases where it tries to upgrade but the dependency resolution fails
that's fine, the problem is that it continues to the next pkg, instead of aborting. maybe a better way would be to have it abort by default, then have another option -f or something to make it ignore the error and continue(unsafe)

the safest way to do it is to actually resolve the dependency order, i have no idea how if pacman even supports simulating an install though. i'll try and find a way to resolve the order tomorrow if i get some time.

Offline

#6 2009-05-17 23:01:34

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

kumyco wrote:

i know, my point was that when you do pacman -S foo, it will resolve the dependencies(all of them) and install everything which sorta goes against what you were trying to do.

yes, it's completely true, and of course it _is_ a BUG.

the safest way to do it is to actually resolve the dependency order, i have no idea how if pacman even supports simulating an install though. i'll try and find a way to resolve the order tomorrow if i get some time.

this is my favourite approach. Right now, I'm googling about "pacman dependency graph".
Or, I could take the list of packages that my package requires, then trying to check if any of these can be installed without any other dependencies (or, take the smaller "dependency cycle") and install it. Then, continue on and on.
This is the safe method that requires less memory at all.
Of course, the "skip dependency" method is even faster, and it will be provided as an option.

BTW, does anyone knows something about python library for accessing pacman? (even easy wrappers on pacman command are ok wink )

EDIT
* -d option added
* -p option added: it just pretends to install

Last edited by BoySka (2009-05-17 23:19:16)

Offline

#7 2009-05-18 06:47:54

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

python binding to libalpm, http://github.com/ornitorrincos/pyalpm/tree/master not sure if it still works though

Offline

#8 2009-05-18 06:51:40

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: pacatatime - Upgrade your sistem one package at a time

BoySka wrote:

this is my favourite approach. Right now, I'm googling about "pacman dependency graph".

pactree in the pacman-contrib package give a plain text dependency graph.   Not sure if it is at all similar to what you want though...

Offline

#9 2009-05-18 07:38:25

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

Allan wrote:
BoySka wrote:

this is my favourite approach. Right now, I'm googling about "pacman dependency graph".

pactree in the pacman-contrib package give a plain text dependency graph.   Not sure if it is at all similar to what you want though...

yep, it's pretty cool, but I need to reimplement it for two reason:
1. it's a bash script: I need to parse its output OR to reimplement it
2. I don't need installed packages

pyalpm doesn't seem to be complete, while http://home.gna.org/libpypac/ and http://www.infolexikon.de/code/pyalpmm/ seems to be pretty mature, but none of them is in AUR.

So, it'll take a while to code it, and probably I'll end up just parsing pacman output sad
BUT I think doing this is quite important, since it allows the user to install packages ONE at a time, safely.

Offline

#10 2009-05-18 08:50:11

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

nice Allan, i never knew that package existed smile

#!/usr/bin/env python

from subprocess import Popen, call, PIPE
import re, sys

DB_PATH = '/var/lib/pacman'
RX_URL = re.compile('.*/(.*)-[\d.]+-\d+-\w+\.pkg\..*')

if call("pacman -Sy", shell=True) != 0:
    sys.exit(1) # exit if we're supposed to abort on errors

pm = Popen("pacman -Sup", stdout=PIPE, shell=True)

print ('\nResolving targets')
targets = {}
for url in pm.stdout:
    url = url.strip()
    mat = RX_URL.search(url)
    if mat:
        targets[mat.group(1)] = url

if pm.wait() != 0:
    sys.exit(1) # exit if we're supposed to abort on errors

# packages that need installing are in targets
for t in targets:
    print (t)

this script shows how you can parse the urls to get the pkgname,
for now at least, you can use

pactree -l -u -s

for each t in targets to get the list of dependencies
then check each one, if it appears in targets install, from the url directly (targets[name]) then del targets[name] to get rid of that entry
a for loop won't work, so use a while len(targets) > 0
----
i have to go revise for exams now, so i can't do much about helping with porting the dependency resolution just yet

Offline

#11 2009-05-18 09:08:15

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

kumyco wrote:
RX_URL = re.compile('.*/(.*)-[\d.]+-\d+-\w+\.pkg\..*')

nice regexp! I'm "splitting" the string instead. Same results (I hope) but regexp is of course much more elegant smile

for now at least, you can use

pactree -l -u -s

nope, pactree doesn't work with not installed packages. I'll have to reimplement it

i have to go revise for exams now, so i can't do much about helping with porting the dependency resolution just yet

well, thank you for your suggestions! when you want, you can fork my git repo at:
http://github.com/boyska/pacatatime/

Offline

#12 2009-05-18 09:45:20

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

just had a thought, do you normally have enough ram to fit the packages ?
if you do you could also create a ramdisk and change the cachedir that
-- i think i know an easy way to get the dependencies without building a tree so that'd be there as well
but if you cold push it into ram, you'd another (total remaining/2)400mb? i think. if it overflows then it gets pushed back to swap (if you have any)

Offline

#13 2009-05-18 10:02:59

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

kumyco wrote:

just had a thought, do you normally have enough ram to fit the packages ?

Well, what I do on my eee is to mount tmpfs over /tmp. I don't want this script to be so invasive, so it could be provided as an option. Anyway, free ram depends on computer type and usage, so we can't _assume_ that everyone has enough free ram.

if you do you could also create a ramdisk and change the cachedir

The cachedir is already changed to a folder under /tmp (that is often a tmpfs, especially on netbooks, and I suggest to do it even on desktops). BUT I want to minimize ram usage, too.

i think i know an easy way to get the dependencies without building a tree

yikes cool! how do you do it? I was thinking about "finding leaves": trying "pacman -Sp <pkg>" for each package, and count the output lines. If there is a SINGLE url, then it's a leaf and we can install it with no deps.
This method *should* work almost always, except for packages that make a "cycle": they will be installed after trying to install all the leafes, together with packages that dependes on one of the packages of the cycle.
Building a tree will allow us to FIND the cycle, and install it; then, the packages that depend on it can be installed one at a time.

but if you cold push it into ram, you'd another (total remaining/2)400mb? i think. if it overflows then it gets pushed back to swap (if you have any)

we can't assume that everyone has a swap: netbooks almost never have, and I don't have it on my desktop, too.
Of course we could create a swap file and tmpfs would automatically use it, but I don't think this is a KISS approach to the problem.
So, I think the user should care about handling his /tmp folder, his ram, his swap. It's just "mount tmpfs -t tmpfs /tmp/that_directory" after all.
If a user has enough ram/swap, he can mount a folder as tmpfs and do "pacman -Syu" with no problems; but we are talking about boxes with hardware limitations.

Offline

#14 2009-05-18 12:07:49

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

#!/usr/bin/env python

from subprocess import Popen, call, PIPE
import re, sys
sys.stdout = sys.stderr

DB_PATH = '/var/lib/pacman'

# extract repo, pkgname, and version
RX_URL = re.compile('.*/(.*?)/os/.*/(.*)-([\d.]+-\d+)-\w+\.pkg\..*', re.UNICODE)

# match the pkgname in dependencies
# are the legal characters in a package name only alphanumeric, _ and - ?
# some dependencies have a version comparison following the name
RX_NAME = re.compile('([\w-]+)', re.UNICODE)

# set this to False to reparse the depends file everytime (save a little memory)
CACHE_DEPENDS = True

# set to false to continue on error .. calls to Abort()
ABORT_ON_ERROR = True

QUIET = False
########################################################################
########################################################################

class NotAUrl(Exception):
    def __init__(self, message):
        self.parameter = message
    
    def __str__(self):
        return repr(self.parameter)

# convenience class to hold repo, name, ver and url
# raises a NotAURL error
class Target(object):
    def __init__(self, url):
        url = url.strip().decode()
        mat = RX_URL.search(url)
        if mat:
            (self.repo, self.name, self.version) = mat.group(1, 2, 3)
            self.url = url
        else:
            raise NotAUrl('Invalid Url: `%s\'' % (url))

def Abort(message='', retcode=1):
    if message and not QUIET:
        print(message)
    if ABORT_ON_ERROR:
        sys.exit(retcode)

# cache the dependencies
dependencies = {}
# read the depends file and return a list of the dependencies, and cache it
def target_depends(target):
    if CACHE_DEPENDS and dependencies.get(target.name):
        return dependencies[target.name]
    
    path = '%s/sync/%s/%s-%s/depends' % (DB_PATH, target.repo, target.name, target.version)
    in_deps = False
    deps = []
    fp = open(path)
    for ln in fp:
        ln = ln.strip()
        if ln:
            if ln[0] == '%':
                if ln == '%DEPENDS%':
                    in_deps = True
                else:
                    break
            else:
                mat = RX_NAME.match(ln)
                if mat:
                    deps.append(mat.group(1))
    fp.close()
    
    if CACHE_DEPENDS:
        dependencies[target.name] = deps
    
    return deps

# do all that needs to be doen in here, at this stage all dependendencies are resolved
# make sure to del targets[target.name] so the target is removed(dep)
def target_install(target):
    # check all the target's deps against targets
    for pkg in target_depends(target):
        dep = targets.get(pkg)
        if dep:
            target_install(dep)
    
    print('\t:: %s ::' % target.name)
    cmd = 'pacman --noconfirm --needed -Sd %s' % (target.name)
    #if call(cmd, shell=True, stdout=sys.stderr) != 0:
        #Abort('Package Installation failed')
    del targets[target.name]

########################################################################
########################################################################



if call('pacman -Sy', shell=True, stdout=sys.stderr) != 0:
    Abort('DB Sync failed')

if not QUIET:
    MSG_RESOLVING = 'Resolving targets ...'
    sys.stderr.write('\n%s Please Wait.' % (MSG_RESOLVING))
    sys.stderr.flush()

pm = Popen('pacman -Sup', stdout=PIPE, shell=True)

targets = {}
for url in pm.stdout:
    try:
        t = Target(url)
        targets[t.name] = t
    except NotAUrl:
        pass

if pm.wait() == 0:
    if not QUIET:
        sys.stderr.write('\r%s Done.         \n' % (MSG_RESOLVING))
else:
    if not QUIET:
        sys.stderr.write('\r%s Fail.         \n' % (MSG_RESOLVING))
    Abort('Getting update list failed')

if targets:
    if not QUIET:
        print('Installing Updates\n')
    
    while targets:
        target = targets[tuple(targets.keys())[0]]
        target_install(target)
        
else:
    print('No updates available.')

it only print the packages it will install, in the correct order ....
the install is commented out because i think pacman has a dependency resolution bug

i'm testing with extra/ffmpeg and extra/xine-lib .. they are to be upgraded to their testing versions
problem, is pacman fails if i try to install one pkg -- with error on xine 'xine-lib: requires ffmpeg>=20081220' .. but if i do pacman -Su it resolves them, and install in the same order i tried .. at this stage i tink -Sd is just as safe, but let's make sure it's not my bug first

-----

about the ramdisk/tmpfs part, i was referring to your particular usage , i personally wouldn't put it in a script, just too many way for it to go wrong

-----

my method is to search the direct dependencies of each package to be installed, i know it is only 1 level deep because pacman has to resolve the deps before it can know the urls.
and we're only interested in packages that are being installed, so we only need track that set of depds

i hope the code is readable enough for you to understand, i'm not very good with documentation so just ask me and i'll explain anything

Last edited by kumyco (2009-05-18 12:13:51)

Offline

#15 2009-05-18 12:37:52

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

thank you for the DB_PATH/sync/.../depends idea, I didn't knew about that, and it will be useful.
Now I implemented a workaround that is pretty good.
I do pacman -Sp <package> for each one, and count the line of output: they are its dependencies!
Then I install the package which has fewer dependencies.
The list is resorted at each cycle, to be sure. This is a bit CPU-expensive, but is simple and seems to work well.
If there are no cycles, than it will manage to install one package at a time.
If there are cycles, they will be installed properly (all and only packages in the cycle will be installed together)
If you can confirm that this work well, it seems to me a pretty good compromise.

Offline

#16 2009-05-18 12:52:00

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

the script on github doesn't work .. checking the number of deps is fine for some packages, but others depend on packages that are being installed, and more. so it doesn't always work.

btw, i used pacman -S instead of pacman -U because the the latter will download the pkg first which might be undesirable in case there are errors, they both have the same effect in this case, in my script saving--self.url is redundant and the file /depends reading can be simplified, as i don't think anything else is %xxx% is allowed in the file, got that mixed up with /desc

Offline

#17 2009-05-21 21:42:33

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

sorry for my late on replying.
I finally had the time to deeply understand your code, and it is an interesting approach.
ATM I'm a bit busy, but I fixed from -U to -S (so better wink ). The dependency resolution is handled in the same way as before, as I found that system "safe": it can't brings to error (-S will keep us safe), and it should catch the optimal resolution order: it is far from elegant and it wastes CPU, anyway.
If there aren't bugs, I'll work on "restyling": I want a clearer separation between "pacman jobs", "graph jobs" and "UI".
So, please give me feedback and I'll start working on wink

Offline

#18 2009-05-22 16:36:48

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

good idea not to use pacman -U, it will download the file into the current directory (prolly a bug) instead of caching it
the error i got from pacman -S wasn't a bug, it was an old pkg i built from aur some time ago that depended on a specific version pf ffmpeg.

the way i did it was actually written in shellscript, but i just can't remember why i switched to python.
maybe it'll come to me after i get some sleep.

Offline

#19 2009-06-11 20:39:29

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

Hi, I want to excuse me for my late on the promised refactoring.
I see that there aren't new messages here, but I hope this doesn't mean there isn't any interest in this little piece of sw.

Well, I have some little good news:
- new pacatatime version checks if the user has root privileges. It's just a 3-lines enhancement, but it was useful for me
- refactoring is going on. you can follow the development at http://github.com/boyska/pacatatime/tree/refactor. I can't promise anything about it because I'm VERY busy with university exam, but I'll try to do something.

Now I have a simple but pretty good object to represent a digraph and to represent a dependency graph. I have also a good, simple algorithm to extract a valid installing sequence.
I just need to code the actual installing/cache cleaning part, then I'm done. wink

Offline

#20 2009-06-12 13:20:28

kumyco
Member
From: somewhere
Registered: 2008-06-23
Posts: 153
Website

Re: pacatatime - Upgrade your sistem one package at a time

i didn't read much into the code, but it appears sometimes (first run) it just exits when there is a pacman error, also you can remove the import to deprecated popen2
apart from that and line 172 just printing a (sometimes empty) list, all looks ok so far, good work
good luck with you exams.

Offline

#21 2009-06-22 22:42:44

BoySka
Member
Registered: 2008-05-09
Posts: 25

Re: pacatatime - Upgrade your sistem one package at a time

BIG UPDATE
Finally, I merged the "refactor" branch into the master one.
I've done a _lot_of_ work on refactoring, code cleaning, fixing and so on, and I hope you'll like the result.

There are lot of changes/fixes/improvements:
* Refactoring: now it's Object Oriented, no useless code, the code is pretty good (I hope)
* Graph: the class PacGraph mantains a dependency graph. This is fundamental to us, but it's also a useful piece of code
* Checks if the user has root privileges, otherwise exits
* Better url parsing: it's now combined with direct db access, to minimize errors. It should now be very safe.
* logging: it logs both to a file (~/.pacatatime.log) and console (only if -v is used).
* -t --show-tree: show dependency tree!
* Checks if pacman is actually installing more than one package at a time: if so, prompt the user (if --batch is not used)

I suggest you all to upgrade:

yaourt -S pacatatime-git

Let me know impression/suggestions

Offline

Board footer

Powered by FluxBB