cower and burp: fast and simple AUR {up,down}loading

falconindy · 2010-05-14 23:11:41

To preface this: Slurpy is awesome. Simple, and does exactly what I want. But being written in Python, it's not as snappy as it could be... So I rewrote it in C. Enter 'cower'.

I've been using cower on a day to day basis with extremely few problems for the past month or so. Lightning fast, and supports nearly every feature Slurpy has. Almost every feature, except uploading... After much procrastination and some legitimate delays due to school, I wrote 'burp' to take care of uploading as I wasn't keen on combining it with cower. I think it's better served as a separate utility as some of the "big guns" like Clyde already download, but don't offer the ability to upload. burp is very light and requires only curl to go about its business.

cower 3.0.1
Usage: cower <operations> [options] target...

 Operations:
  -d, --download          download target(s) -- pass twice to download AUR dependencies
  -i, --info              show info for target(s) -- pass twice for more detail
  -m, --msearch           show packages maintained by target(s)
  -s, --search            search for target(s)
  -u, --update            check for updates against AUR -- can be combined with the -d flag

 General options:
  -f, --force             overwrite existing files when downloading
  -h, --help              display this help and exit
      --ignore <pkg>      ignore a package upgrade (can be used more than once)
      --ignorerepo <repo> ignore a binary repo (can be used more than once)
      --nossl             do not use https connections
  -t, --target <dir>      specify an alternate download directory
      --threads <num>     limit number of threads created
      --timeout <num>     specify connection timeout in seconds

 Output options:
  -c, --color[=WHEN]      use colored output. WHEN is `never', `always', or `auto'
      --debug             show debug output
      --format <string>   print package output according to format string
      --listdelim <delim> change list format delimeter
  -q, --quiet             output less
  -v, --verbose           output more

cower
cower-git

2.0 is a fairly major rewrite, sporting yajl in place of jansson for JSON parsing.

burp 1.6.1
Usage: burp [options] targets...

 Options:
  -h, --help                Shows this help message.
  -u, --user                AUR login username.
  -p, --password            AUR login password.
  -c CAT, --category=CAT    Assign the uploaded package with category CAT.
                              This will default to the current category
                              for pre-existing packages and 'None' for new
                              packages. -c help will give a list of valid
                              categories.
  -C FILE, --cookies=FILE   Use FILE to store cookies rather than the default
                              temporary file. Useful with the -k option.
  -k, --keep-cookies        Cookies will be persistent and reused for logins.
                              If you specify this option, you must also provide
                              a path to a cookie file.
  -v, --verbose             be more verbose. Pass twice for debug info.

  burp also honors a config file. See burp(1) for more information.

burp
burp-git

Each includes a man page with somewhat more detail about their respective operation.

Big thanks to:
- the Pacman crew for maintaining insanely readable code which I learned and borrowed from tremendously.
- rson and the rest of the Slurpy crew for providing the inspiration for both of these utils.
- fogobogo for trying to break cower and for giving me enough prodding to finally write burp.

3/23: update usage snapshots

Last edited by falconindy (2015-01-08 19:44:42)

evr · 2010-05-14 23:46:05

very nice! i just tried out burp and it seems to be really light and fast. i'll play around with it more later.

Daenyth · 2010-05-14 23:48:02

Nice! I had been planning to make a little wrapper script around makepkg --source && aurploder, but I'll have to look at burp now.

falconindy · 2010-05-15 21:17:08

Fixed a pair of bugs and tagged release 1.1 of burp.

destruct · 2010-05-15 22:37:15

Great work. Its indeed very fast

Stefan Husmann · 2010-05-16 05:26:06

I looked for something like burp long time. aurploader is nice, but too interactive. Thank you.

falconindy · 2010-05-16 20:32:39

Added the option to keep persistent cookies in burp-git. This can be used in place of a username/password login.

falconindy · 2010-05-21 02:18:11

Tagged cower 2.0. The major change is that I ditched jansson in favor of yajl which means no AUR dependencies. Seems to be just as fast and, thanks to yajl, I was able to do some major code refactoring and sneak in some mild improvements/refinements.

faelar · 2010-05-21 12:26:43

Thank you very much for cower, here a quick "time X -s media" test, results are impressives :

slurpy :
real    0m1.656s
user    0m0.369s
sys    0m0.108s

cower :
real    0m0.769s
user    0m0.019s
sys    0m0.009s

faelar · 2010-06-23 13:30:59

I think you've already spotted this : pacman upgrade has broken cower.

cower: error while loading shared libraries: libalpm.so.4: cannot open shared object file: No such file or directory

I also add a feature request. I know you don't want to bloat your tool, but I would like to know if it is possible to change the output of cower -u. I don't know if you have tried or used aurcheck, but I really like how it displays aur packages to update and the current version of each package. One downside of aurcheck is that it's painfully slow... Please tell me if you like this idea or not, and thank you again for cower, I really like it !

falconindy · 2010-06-23 13:47:25

1) You need to rebuild cower so that it properly links against the new soname. Standard fare for any soname bump.
2) cower already does this. The only reason it wouldn't show the local and remote package versions is because you've used the -q flag with -u.

faelar · 2010-06-23 16:07:50

1) Thank you it's working fine again.
2) Yes cower already show package version but that's not what I'm asking for. I think it will be easier to show you a screenshot of aurcheck : Screenshot.
Sorry I don't have any pkg to update but that looks like the todo package on the screenshot, with version numbers in green instead of red.

Last edited by faelar (2010-06-23 16:08:45)

falconindy · 2010-06-23 16:15:33

Without and with verbosity:
vNHE2cg
Is this not what you're looking for?

andresp · 2010-06-23 16:27:41

Awesome. cower is the only decent aur client.

Every other one ive tried is juggling 20 tasks at once!

faelar · 2010-06-23 17:14:29

@falconindy : Oups, I missed the -v ... ok so cower is awesome as it is, thank you and sorry if I wasted your time

falconindy · 2010-11-07 03:02:25

So, I'm working on a multi threaded rewrite of cower. Yes, it's overkill. Yes, I'm probably insane. However, as with the original write, it served its purpose and I learned what I wanted to. I've been able to add a few improvements as well:

- True recursive dependency downloading (instead of the current single descent)
- Selective upgrade checking (e.g. cower -u pkg1 pkg2)

On top of this, I'm much happier with the code base -- for more or less the same punch, its 2/3's the raw LOC (1700 versus 2700).

Results so far are amusing:

$ { time cower -sq svn bzr git darcs ab; } | wc -l

real    0m1.275s
user    0m0.085s
sys    0m0.000s
5616

$ pacman -Qqm | tee >(time cower -u) >(wc -l) >/dev/null
36
real    0m0.837s
user    0m0.308s
sys    0m0.004s

$ time cower -dd haskell-cabal
:: haskell-cabal downloaded to /home/noclaf/tmp
warning: ghc is available in extra
:: haskell-array downloaded to /home/noclaf/tmp
:: haskell-containers downloaded to /home/noclaf/tmp
:: haskell-directory downloaded to /home/noclaf/tmp
:: haskell-filepath downloaded to /home/noclaf/tmp
:: haskell-old-time downloaded to /home/noclaf/tmp
:: haskell-old-locale downloaded to /home/noclaf/tmp
:: haskell-unix downloaded to /home/noclaf/tmp
:: haskell-pretty downloaded to /home/noclaf/tmp
:: haskell-process downloaded to /home/noclaf/tmp

real    0m2.730s
user    0m0.314s
sys    0m0.008s

$ time cower -d {systemd,burp,cower,slurpy}{,-git} kernel26-{ck,zen}
:: burp-git downloaded to /home/noclaf/tmp
:: cower downloaded to /home/noclaf/tmp
:: cower-git downloaded to /home/noclaf/tmp
:: systemd-git downloaded to /home/noclaf/tmp
:: burp downloaded to /home/noclaf/tmp
:: slurpy downloaded to /home/noclaf/tmp
:: slurpy-git downloaded to /home/noclaf/tmp
:: systemd downloaded to /home/noclaf/tmp
:: kernel26-zen downloaded to /home/noclaf/tmp
:: kernel26-ck downloaded to /home/noclaf/tmp

real    0m0.885s
user    0m0.309s
sys    0m0.003s

Recursive depends take a while because they're not threaded (yet?).

For any AUR maintainers reading this: concurrent connections are capped at 10. I can reel this back in if desired.

For anyone who's interested in helping me test, here's the good news is: it's nearly done and I'd be thrilled if anyone could help me break this. I need to work on the output a bit -- color is yet to be implemented. However, I believe it's otherwise feature complete compared to the current stable release. The bad news: it requires xavier's branch of pacman with the pactree branch merged. If you're feeling brave, feel free to grab it from my experimental branch on github. If git makes you queasy but you'd like to try this out, feel free to find me on IRC and I can give you a hand.

JokerBoy · 2010-11-12 00:13:22

i really like this app, but.. there is a problem..

$ cower --color=always -vd mime-archpkg
INTERNAL ERROR: Function 'archive_read_open' invoked with archive structure in state 'closed', should be in state 'new'
zsh: abort cower --color=always -vd mime-archpkg
$ pacman -Q cower
cower 2.5.3-2

falconindy · 2010-11-12 00:29:28

this is fixed in cower-git.

JokerBoy · 2010-11-12 00:59:44

hmm, indeed, but in cower-git the verbose flag is missing.

if i can make a little suggestion for cower -vu, it would be great to output if the package checked is flagged out dated

falconindy · 2010-11-12 01:04:49

I've ditched the verbose flag because it was really only used for update and I never personally used it beyond testing. You can get more or less the same info now from --debug.

.:B:. · 2010-11-12 07:17:26

I'm gonna try this as well. I love slurpy but yes, it's python .

falconindy · 2010-11-13 04:12:40

cower-git is now multi threaded. It builds with a small kludge until pacman 3.5. The results when dealing with dependencies won't be as accurate as they will be come pacman 3.5, but they aren't any less accurate than they were previously. You can, of course, build against pacman-git instead and get this minor benefit now.

Be aware that I've ditched the config file for customized color. It adds a level of complexity to the code I'm not happy with, and for not enough gain.

usage looks very similar:

cower 2.4-52-g9eb8387
Usage: cower <operations> [options] PACKAGE...

 Operations:
  -d, --download          download PACKAGE(s) -- pass twice to download AUR dependencies
  -i, --info              show info for PACKAGE(s) -- pass twice for more detail
  -s, --search            search for PACKAGE(s)
  -u, --update            check for updates against AUR -- can be combined with the -d flag

 General options:
  -c, --color[=WHEN]      use colored output. WHEN is `never', `always', or `auto'
      --debug             show debug output
  -f, --force             overwrite existing files when downloading
  -h, --help              display this help and exit
      --ignore <pkg>      ignore a package upgrade (can be used more than once)
  -q, --quiet             output less
      --ssl               create connections over https
  -t, --target <dir>      specify an alternate download directory

Last edited by falconindy (2010-11-13 04:13:26)

Sara · 2010-11-13 05:12:03

I'm already preferring this to Packer. Thanks .

One question though. I edited the source code to use BOLDBLACK to display packages (which is defined in cower.c), but it displays as a dark grey. Packer's and pacman's boldblack really is black, though. (I'm using cower-git, hence needing to edit the source code to change the colors). I'm using a white terminal, hence, sticking with BOLDWHITE is rather impossible.

Last edited by Sara (2010-11-13 05:53:39)

faelar · 2010-11-17 16:59:43

I've ditched the verbose flag because it was really only used for update and I never personally used it beyond testing. You can get more or less the same info now from --debug.

I used it everytime, I like to see which packages were taken from AUR when updating. If you don't mind, I'd like to ask you to add this one back. If you prefer not to (it is your code after all), may I suggest you to at least add a quicker way to call the debug output ? cower -ud --debug is more annoying to type than cower -uvd.

I also encounter a small bug. In my bashrc I've got :
alias cower="cower -t ~/Downloads"
When I launch "cower -s pacakge" it works fine, but when I update with "cower -ud" packages are downloaded to the current directory instead of Downloads. Prior to cower's update it worked well.

Anyway, I'm looking forward to the next awesome cower version !

falconindy · 2010-11-17 17:13:54

faelar wrote:

I used it everytime, I like to see which packages were taken from AUR when updating. If you don't mind, I'd like to ask you to add this one back. If you prefer not to (it is your code after all), may I suggest you to at least add a quicker way to call the debug output ? cower -ud --debug is more annoying to type than cower -uvd.

This sounds reasonable. I can link up -D (or similar) to --debug or reimplement -v if I can find a way to do it that isn't tedious. I have a few ideas.

faeler wrote:

I also encounter a small bug. In my bashrc I've got :
alias cower="cower -t ~/Downloads"
When I launch "cower -s pacakge" it works fine, but when I update with "cower -ud" packages are downloaded to the current directory instead of Downloads. Prior to cower's update it worked well.

Bug reports are awesome, but I need to know what version you're using that this no longer works in. I don't recall touching any code related to this recently.

edit: cannot reproduce on the latest tag (2.5.4).

$ alias cow='cower -t ~/tmp'
$ cow -ud
:: freetype2-infinality downloaded to /home/noclaf/tmp
:: ncmpcpp-git downloaded to /home/noclaf/tmp

faeler wrote:

Anyway, I'm looking forward to the next awesome cower version !

3.0 will be tagged when pacman 3.5 is moved to core due to some code decisions I've made. Anyone who wants to be an early adopter and tell me my code sucks is highly encouraged to do so.

Last edited by falconindy (2010-11-17 23:39:57)

Arch Linux

#1 2010-05-14 23:11:41

cower and burp: fast and simple AUR {up,down}loading

#2 2010-05-14 23:46:05

Re: cower and burp: fast and simple AUR {up,down}loading

#3 2010-05-14 23:48:02

Re: cower and burp: fast and simple AUR {up,down}loading

#4 2010-05-15 21:17:08

Re: cower and burp: fast and simple AUR {up,down}loading

#5 2010-05-15 22:37:15

Re: cower and burp: fast and simple AUR {up,down}loading

#6 2010-05-16 05:26:06

Re: cower and burp: fast and simple AUR {up,down}loading

#7 2010-05-16 20:32:39

Re: cower and burp: fast and simple AUR {up,down}loading

#8 2010-05-21 02:18:11

Re: cower and burp: fast and simple AUR {up,down}loading

#9 2010-05-21 12:26:43

Re: cower and burp: fast and simple AUR {up,down}loading

#10 2010-06-23 13:30:59

Re: cower and burp: fast and simple AUR {up,down}loading

#11 2010-06-23 13:47:25

Re: cower and burp: fast and simple AUR {up,down}loading

#12 2010-06-23 16:07:50

Re: cower and burp: fast and simple AUR {up,down}loading

#13 2010-06-23 16:15:33

Re: cower and burp: fast and simple AUR {up,down}loading

#14 2010-06-23 16:27:41

Re: cower and burp: fast and simple AUR {up,down}loading

#15 2010-06-23 17:14:29

Re: cower and burp: fast and simple AUR {up,down}loading

#16 2010-11-07 03:02:25

Re: cower and burp: fast and simple AUR {up,down}loading

#17 2010-11-12 00:13:22

Re: cower and burp: fast and simple AUR {up,down}loading

#18 2010-11-12 00:29:28

Re: cower and burp: fast and simple AUR {up,down}loading

#19 2010-11-12 00:59:44

Re: cower and burp: fast and simple AUR {up,down}loading

#20 2010-11-12 01:04:49

Re: cower and burp: fast and simple AUR {up,down}loading

#21 2010-11-12 07:17:26

Re: cower and burp: fast and simple AUR {up,down}loading

#22 2010-11-13 04:12:40

Re: cower and burp: fast and simple AUR {up,down}loading

#23 2010-11-13 05:12:03

Re: cower and burp: fast and simple AUR {up,down}loading

#24 2010-11-17 16:59:43

Re: cower and burp: fast and simple AUR {up,down}loading

#25 2010-11-17 17:13:54

Re: cower and burp: fast and simple AUR {up,down}loading

Board footer