pywer - A new library based AUR helper written in python

KaiSforza · 2013-09-29 17:14:20

At the start of September (2013/09/02) I started working on a new python AUR helper. Previously I had been using cower, and was actually really happy with it. pywer[1][2], as the name implies, is a python version of cower. Currently it has most of the features, but is still missing some very important ones such as downloading dependencies. I would really like to get some feedback and possibly patches for this to imporove it.

One thing that makes it stand out against some other AUR helpers is that instead of sending multiple requests to the AUR server (which can be specified easily) to get information on more than one package (be it for info or checking for updates), instead it uses the multiinfo request type, meaning that we can check for updates much faster than cower.

Also, as you can see in the title, it is a library based AUR helper. So instead of writing one massive python script to do everything, I've written a module, "libaur", which contains submodules for printing, communicating with the rpc and other such things. It shouldn't be too hard to use libaur in your own scripts, and I've written documentation[3] for just about everything that is updated to the git HEAD whenever I push to my server.

As I said, any contribution is welcome, be it bug reports or patches. Or both. Right now I am working towards a 1.0 release, but there are still quite a few things I need to mark off the TODO list before then.

[1] personal server: http://git.kaictl.net/wgiokas/pywer.git/
[2] github: https://github.com/KaiSforza/pywer
[3] documentation: http://kaictl.net/docs/pywer/
AUR package: https://aur.archlinux.org/packages/python-pywer-git/

Thanks for reading! I hope someone finds some use for this project.

EDIT: If you do end up using this, beware that it is still heavily in development, and as such things may change a lot till the 1.0 release.

Last edited by KaiSforza (2013-09-29 19:16:55)

tomk · 2013-09-29 17:34:20

An amusing bit of history, in case you weren't already aware:

rson451 wrote:

slurpy is another AUR helper script written in Python.

https://bbs.archlinux.org/viewtopic.php?id=75984

falconindy wrote:

Slurpy is awesome. Simple, and does exactly what I want. But being written in Python, it's not as snappy as it could be... So I rewrote it in C. Enter 'cower'.

https://bbs.archlinux.org/viewtopic.php?id=97137

Spyhawk · 2013-09-29 17:45:16

tomk> Mycomment would have been similar to yours if you hadn't posted before

KS> I'm not sure that the multiinfo method is really that faster, since cower is multithreaded anyway. Any benchmark done on your side?

Last edited by Spyhawk (2013-09-29 17:53:21)

KaiSforza · 2013-09-29 17:57:48

tomk wrote:

An amusing bit of history, in case you weren't already aware:
rson451 wrote:
slurpy is another AUR helper script written in Python.
https://bbs.archlinux.org/viewtopic.php?id=75984
falconindy wrote:
Slurpy is awesome. Simple, and does exactly what I want. But being written in Python, it's not as snappy as it could be... So I rewrote it in C. Enter 'cower'.
https://bbs.archlinux.org/viewtopic.php?id=97137

I had learned about this a few weeks into writing pywer. I was amused. But the one thing that cower doesn't do is contain a reusable library (which is possible in C, but I feel like people will be happy to have a scripting interface to the AUR). That's mostly why I was writing it. Originally it was just the library, really, and I steadily tacked on bits that let you actually use it from the terminal. Otherwise, cower is perfectly fine. Also, pywer can be used on non-Archlinux platforms with ease for searching, downloading, and getting information (the --update option requires pyalpm, though)

Spyhawk wrote:

KS> I'm not sure that the multiinfo method is really that faster, since cower is multithreaded anyway. The difference might be not significative... any benchmark done on your side?

I am quite sure. With multiinfo, you are still sending a single request, whereas with cower and using more packages than you can threads (my internet limits me to 2 threads) it's significantly slower. I would also argue that cower uses more bandwidth, but we're not in the dial-up age anymore, so that argument is pretty worthless.

Spyhawk · 2013-09-29 22:09:59

KaiSforza wrote:

I am quite sure. With multiinfo, you are still sending a single request, whereas with cower and using more packages than you can threads (my internet limits me to 2 threads) it's significantly slower. I would also argue that cower uses more bandwidth, but we're not in the dial-up age anymore, so that argument is pretty worthless.

Well, with a 2 threads limit you might feel the difference, but I don't think this is representative of the majority of people. I'm using the 10 threads default setting, and I'm sure I could increase that limit a bit. Anyway, pywer seems to be a fairly interesting project

KaiSforza · 2013-09-30 02:42:44

Spyhawk wrote:

Well, with a 2 threads limit you might feel the difference, but I don't think this is representative of the majority of people. I'm using the 10 threads default setting, and I'm sure I could increase that limit a bit. Anyway, pywer seems to be a fairly interesting project :)

That, honestly, is not the main intention though. It's just a side effect of writing this after cower. I'm pretty sure type=multiinfo is quite new. I really wrote this for the library so others could use it if they're making their own AUR helper, so as not to duplicate work.

Xyne · 2013-09-30 03:39:51

KaiSforza wrote:

That, honestly, is not the main intention though. It's just a side effect of writing this after cower. I'm pretty sure type=multiinfo is quite new. I really wrote this for the library so others could use it if they're making their own AUR helper, so as not to duplicate work.

python3-aur has supported multiinfo since it first became available. I have not looked at your code, but I suspect that it duplicates much of my previous work.

KaiSforza · 2013-09-30 04:07:33

I looked at python3-aur, and was somewhat saddened that it did caching I think this is not a bad idea, but not in the way you implemented it (As part of the AUR class). The caching that is done is completely unneeded, and I would actually prefer not to use it for the sake of simplicity. It is also lacking quite a few things:

* Plain, simple search (like pacman -{Q,S}s)
* Separated printing and querying code (these should not be in the same function)
* Color
* Any kind of safe PKGBUILD parsing
* Ability to query installed packages without pacman or subprocesses
* Downloading

Some things that, after looking through your code, I am interested in are the uploader and other non-query parts of python3-aur. That's just what I see from a cursory glance at things, though if there is more to your code than meets the eye, please bring it up. (Also, the PKGBUILD on the AUR for it fails if you're in a virtualenv because --prefix is not set.)

(I had actually not seen this at all when I started writing pywer.)

Xyne · 2013-09-30 05:48:01

Caching is entirely configurable with the ttl parameter. In most cases, package data does not constantly change, so temporarily caching the data makes sense (I think the default is 5 or 15 minutes).

Plain/simple search is easily configurable. The provided aurquery tool is simply a demonstration of what can be done. it would be trivial to modify it. The library is independent of aurquery.

Color can easily be added in a client. It has no business being in the backend.

There is no such thing as complete and safe PKGBUILD parsing, nor is that really something that belongs in an AUR library. That should be in a separate library (or module at least).

The code uses pyalpm to query installed packages if necessary. Why are you suggesting otherwise? Am I missing something?

Downloading is trivial and also better left to the client.

KaiSforza · 2013-09-30 06:40:34

Xyne wrote:

Caching is entirely configurable with the ttl parameter. In most cases, package data does not constantly change, so temporarily caching the data makes sense (I think the default is 5 or 15 minutes).

Okay, that's fine. I may eventually try something of this nature, but it would make more sense for this to be disabled by default, just for the sake of not writing anything unneeded to the system. (Also, TTL seems more appropriate for network packets, not for a database entry. Maybe 'cache_time' would be better?)

Plain/simple search is easily configurable. The provided aurquery tool is simply a demonstration of what can be done. it would be trivial to modify it. The library is independent of aurquery.

aurquery simply runs `run_main()`...so it would have to take place in the library...

Color can easily be added in a client. It has no business being in the backend.

True. And pywer is a client and a library, just that the client is very tightly knit with the library.

There is no such thing as complete and safe PKGBUILD parsing, nor is that really something that belongs in an AUR library. That should be in a separate library (or module at least).

You don't need complete parsing, just enough to get the variables that are there, such as "depends" and "makedepends". libaur does this without executing any code from the PKGBUILD, so I would call that quite safe. (This is going to be the foundation for dependency resolution when downloading)

The code uses pyalpm to query installed packages if necessary. Why are you suggesting otherwise? Am I missing something?

I don't see any use of pyalpm in your code. Currently it just calls `pacman -Qqm` for foriegn packages. It seems like you're trying to avoid pyalpm in this comment.

python3-aur-2013.8 % grep -R pyalpm *                                                                                                                                                                               
aurtomatic:# Avoids the dependency on pyalpm and is probably simpler programmatically.
python3-aur-2013.8 %

Downloading is trivial and also better left to the client.

This is a client and a library, so it needs to do both.
(also, is there any VCS repo with this codes history?)

And on that note:

I do really like python3-aur, I have nothing against it, and will probably use it for inspiration in the future. However, pywer does slightly different things, with a subset of our libraries overlapping. I am still somewhat new to python, and my code could use a lot of work, but that's really what this project is for.

Last edited by KaiSforza (2013-09-30 06:41:22)

Xyne · 2013-09-30 23:39:08

KaiSforza wrote:

Okay, that's fine. I may eventually try something of this nature, but it would make more sense for this to be disabled by default, just for the sake of not writing anything unneeded to the system. (Also, TTL seems more appropriate for network packets, not for a database entry. Maybe 'cache_time' would be better?)

It's funny that you raise this point as I am usually very adamant about avoiding disk IO when I can. My motivation here is to reduce server load as much as possible, given how many redundant queries people likely run within that caching window. Regardless, anyone can write a client that disables the caching.

As for the name, ttl may not be the best, but it's not ambiguous in this context either, so I think I will leave it.

KaiSforza wrote:

Xyne wrote:
Plain/simple search is easily configurable. The provided aurquery tool is simply a demonstration of what can be done. it would be trivial to modify it. The library is independent of aurquery.
aurquery simply runs `run_main()`...so it would have to take place in the library...

Sorry, my wording was poor. I do not mean that it is trivial to change aurquery itself. I mean that aurquery is itself trivial and easy to duplicate and customize with very little code in a separate script.

The main function is only 14 lines long and only really calls 3 very short functions for printing the results of the 3 different accepted queries. It's only in the library itself because I like the library to do something on it's own when run as __main__, but no one is obligated to call that function. The purpose of the library is to provide the interfacing functions and datatype conversions. Please do not focus so much on one simple usage example.

KaiSforza wrote:

There is no such thing as complete and safe PKGBUILD parsing, nor is that really something that belongs in an AUR library. That should be in a separate library (or module at least).
You don't need complete parsing, just enough to get the variables that are there, such as "depends" and "makedepends". libaur does this without executing any code from the PKGBUILD, so I would call that quite safe. (This is going to be the foundation for dependency resolution when downloading)

You do need complete parsing to handle all cases.

I do have experience with this. I used to maintain a tool called bauerbill, which was an AUR-enhanced version of powerpill (dependency resolution, parallel downloads of all packages and source files), and quite popular at one point. It was written in Perl and included a PKGBUILD parsing module that used tricks such as wrapping PKGBUILDs in functions to source them and format them through the output of "set" before parsing them with regexes. It worked quite well most of the time but there were frequent edge cases (conditional blocks, "clever" use of Bash, nested variables (e.g. split packages)). Short of a complete parser that can generate an AST, anything will fail occasionally.

At the same time, if it handles most cases then it is "good enough", but keeping it "good enough" involved constantly chasing down prominent edge cases.

That said, a PKGBUILD parser would be of general utility and deserve its own library. If you decide to implement one, I hope that you will split it from pywer. Given my previous experience with PKGBUILD parsing, I would personally opt for writing a proper parser with something like Haskell's Parsec rather than go down the regex route again, but the task is quite large if you want to handle all the key features such as string operations.

KaiSforza wrote:

The code uses pyalpm to query installed packages if necessary. Why are you suggesting otherwise? Am I missing something?
I don't see any use of pyalpm in your code. Currently it just calls `pacman -Qqm` for foriegn packages. It seems like you're trying to avoid pyalpm in this comment.
python3-aur-2013.8 % grep -R pyalpm *                                                                                                                                                                               
aurtomatic:# Avoids the dependency on pyalpm and is probably simpler programmatically.
python3-aur-2013.8 %

Yep, you're right. I should have looked through it before commenting. Note, however, that the call is made in aurotmatic, which is, again, a demonstration. The call is used to avoid the additional dependency of pyalpm and is completely independent of the library. So far it seems that all of your criticism is against the demonstration tools that I have written mostly as an afterthought, rather than against the library itself.

KaiSforza wrote:

Downloading is trivial and also better left to the client.
This is a client and a library, so it needs to do both.
(also, is there any VCS repo with this codes history?)

pbget is a client that downloads PKGBUILDs. It includes the downloading code. The AUR.RPC module is only for querying information. While it could be useful to complete another module with download functions, that would be generic and better off in a separate package imo.

I have private Git repos but I occasionally commit the sin of rewriting history. Besides, making the VCS available encourages contributions, and once you accept someone's contribution you lock your own code into a given license.

KaiSforza wrote:

And on that note:
I do really like python3-aur, I have nothing against it, and will probably use it for inspiration in the future. However, pywer does slightly different things, with a subset of our libraries overlapping. I am still somewhat new to python, and my code could use a lot of work, but that's really what this project is for.

Don't get me wrong. I'm not trying to stop you. Doing things your own way is both a learning experience and results in something that you fully understand yourself. I simply reacted to your comment about your tool uniquely using multiinfo followed by

KaiSforza wrote:

I really wrote this for the library so others could use it if they're making their own AUR helper, so as not to duplicate work.

From my perspective, re-implementing a wrapper around the RPC interface is a duplication of work and I was disappointed to see it as I think AUR.RPC works very well. All the functionality that you need is there, configurable as you wish, yet you seem to be put off by the included scripts, which are really quite separate imo.

Sorry for going off-topic in your thread, but I really felt the need to reply to the comment.

sas · 2013-10-01 15:28:29

Xyne wrote:

KaiSforza wrote:
You don't need complete parsing, just enough to get the variables that are there, such as "depends" and "makedepends". libaur does this without executing any code from the PKGBUILD, so I would call that quite safe. (This is going to be the foundation for dependency resolution when downloading)
You do need complete parsing to handle all cases.

+1

There are all kinds of non-trivial bash'isms out there in AUR PKGBUILDS, for example you may find stuff like:

case $CARCH in
    i686)   depends=(libgl gtk) ;;
    x86_64) depends=(lib32-libgl lib32-gtk) ;;
esac

or

[ $CARCH == i686 ] && { _arch=32; _md5='9f27516abd229dfsi38swa3cf31f2b43'; } \
                   || { _arch=64; _md5='s832jdh335bd5b959baa1e1add89ca51'; }

source=("Foo-${pkgver}_${_arch}bit.tar.gz")
md5sums=("$_md5")

or

pkgname=perl-foo
_author=JOHNDOE
pkgver=1.0

_module=${pkgname#perl-}
source=("http://search.cpan.org/CPAN/authors/id/${_author:0:1}/${_author:0:2}/$_author/${_module^}-${pkgver}.tar.gz") 
# ^ produces: http://search.cpan.org/CPAN/authors/id/J/JO/JOHNDOE/Foo-1.0.tar.gz

Last edited by sas (2013-10-01 15:30:05)

progandy · 2013-10-01 15:47:01

sas wrote:

Xyne wrote:
KaiSforza wrote:
You don't need complete parsing, just enough to get the variables that are there, such as "depends" and "makedepends". libaur does this without executing any code from the PKGBUILD, so I would call that quite safe. (This is going to be the foundation for dependency resolution when downloading)
You do need complete parsing to handle all cases.
+1

You might be interested in libbash which provides ANTLR grammar for bash http://dev.gentoo.org/~qiaomuf/libbash.html
If you want to create something in python you can start with pyparsing I guess. antlr v3 for python3 is still in development or dead.

Last edited by progandy (2013-10-01 15:50:08)

KaiSforza · 2013-10-01 16:26:09

progandy wrote:

sas wrote:
Xyne wrote:
You do need complete parsing to handle all cases.
+1
You might be interested in libbash which provides ANTLR grammar for bash http://dev.gentoo.org/~qiaomuf/libbash.html
If you want to create something in python you can start with pyparsing I guess. antlr v3 for python3 is still in development or dead.

libaur.PKGBUILD's parsing is really just an expansion of cower's pkgbuild_get_extinfo function. cower only looks for the *depends, provides and conflicts arrays, whereas libaur finds every official variable and uses that when doing the very simple bash substitution (really just replacing $foo or ${foo}, not ${foo[2]} or anything). Not perfect, but it does its job moderatly well.

In the future, there may be another library outside of libaur that does more than libaur, but as it stands this is acceptable for most things. Corner cases, variable expansion (which may be added soon) and crazy $CARCH dependent things aside, libaur will give you most of the important data from a PKGBUILD.

If you want, you can go and look at the test I run for PKGBUILD parsing and start adding on corner cases and getting them to work. I'll accept contributions that make libaur or pywer better!

sas · 2013-10-02 12:28:11

progandy wrote:

You might be interested in libbash which provides ANTLR grammar for bash http://dev.gentoo.org/~qiaomuf/libbash.html

That's pretty cool.
However it looks like it was a 2012 Google Summer of Code project, and the linked git repository didn't get any more commits after that. Do you know what became of the project? Is Gentoo actually using this now, or planning to do so?

KaiSforza wrote:

crazy $CARCH dependent things aside

It's not that crazy, in fact it's a necessity for many AUR packages that deal with pre-compiled binaries (including, but not limited to, proprietary software and games), where sometimes 32bit/64bit versions are bundled separately by upstream (in which case source and md5sums need to be $CARCH dependent), or - more commonly - only a 32bit version exists (in which case depends and optdepends need to be $CARCH dependent).

Last edited by sas (2013-10-02 12:31:04)

KaiSforza · 2013-10-02 18:17:56

I am still hesitant to support every single bashism. Right now I am debating including a less secure parser that simply subprocesses to bash and gets everything that's needed.

Spyhawk · 2013-10-02 22:24:00

Security or efficiency - you actually can't have both. The only way to solve this issue is by parsing the PKGBUILD at source tarballing time (makepkg --source), writing these value in an auxiliary file in tho tarball (.SRCINFO) and making the AUR able to retrieve this information (see FS#16394 in the bugtracker). Unfortunately, this seemingly won't be implemented in a near future.

Arch Linux

#1 2013-09-29 17:14:20

pywer - A new library based AUR helper written in python

#2 2013-09-29 17:34:20

Re: pywer - A new library based AUR helper written in python

#3 2013-09-29 17:45:16

Re: pywer - A new library based AUR helper written in python

#4 2013-09-29 17:57:48

Re: pywer - A new library based AUR helper written in python

#5 2013-09-29 22:09:59

Re: pywer - A new library based AUR helper written in python

#6 2013-09-30 02:42:44

Re: pywer - A new library based AUR helper written in python

#7 2013-09-30 03:39:51

Re: pywer - A new library based AUR helper written in python

#8 2013-09-30 04:07:33

Re: pywer - A new library based AUR helper written in python

#9 2013-09-30 05:48:01

Re: pywer - A new library based AUR helper written in python

#10 2013-09-30 06:40:34

Re: pywer - A new library based AUR helper written in python

#11 2013-09-30 23:39:08

Re: pywer - A new library based AUR helper written in python

#12 2013-10-01 15:28:29

Re: pywer - A new library based AUR helper written in python

#13 2013-10-01 15:47:01

Re: pywer - A new library based AUR helper written in python

#14 2013-10-01 16:26:09

Re: pywer - A new library based AUR helper written in python

#15 2013-10-02 12:28:11

Re: pywer - A new library based AUR helper written in python

#16 2013-10-02 18:17:56

Re: pywer - A new library based AUR helper written in python

#17 2013-10-02 22:24:00

Re: pywer - A new library based AUR helper written in python

Board footer