web scraping + AUR = fun

phrakture · 2005-04-29 19:31:43

seeing as cvsup isn't working with aur yet, I made a little script to download the entire set of aur package directories... yeah, it takes a long time... it's using HTTP and scraping apache's directory listing format... but it works...

anyway, here we go - feel free to do what you want:

# !/bin/sh
# AUR Web Scraping to get all PKGBUILDs
# Aaron Griffin [phrakture]

BASEDIR="$HOME/aur"
PKGURL="http://aur.archlinux.org/packages/"
PKGFILE="index.html"

#get_dir http://www.xyz.com/a
# This function will get all files
# listed in an apache formatted directory list
function get_dir()
{
   local thisdir=`basename $1`

   if [ "x$thisdir" != "x" ]; then
      mkdir $thisdir
      cd $thisdir
      wget -q $1
      if [ $? -eq 0 ]; then
         local files=`grep "[   ]" $PKGFILE |
                      sed 's@.*href="(.*)".*@1@g'`
         #skip parent dir, infinate recursion
         local dirs=`grep "[DIR]" $PKGFILE |
                     grep -v "Parent Directory" |
                     sed 's@.*href="(.*)".*@1@g'`
         rm $PKGFILE

         for f in $files; do
            echo "downloading $thisdir::$f"
            wget -q $1$f
         done

         for d in $dirs; do
            get_dir $1$d
         done

         cd ..
      else
         echo "error downloading directory list : $1"
      fi
   else
      echo "usage: get_dir <apache url>"
   fi
}

cd $BASEDIR
[ -f $PKGFILE ] && rm -f $PKGFILE

get_dir $PKGURL

and don't expect jesus to come out of this and make everything fine and dandy... it's just a script to download the PKGBUILD in whatever format/state they're in on the server

JGC · 2005-04-29 21:14:51

You know you can use wget --mirror for this?

Hmm, --mirror is only useful if you're sure you can exclude stuff from it. Having complete packages isn't very useful

IceRAM · 2005-04-29 21:19:38

wget --mirror copies index.html files as well, not to mention that it goes over index.html?C=X (sorted lists).

phrakture · 2005-04-29 22:08:27

yeah, doesn't mirror grab everything including html files (and I'd assume it's might follow the "parent directory" links)

this could also be extended to get one package at a time... *shrug*

lazychris2000 · 2005-06-23 02:50:26

I ran the script, now I have alot of packages/pkgbuilds in my home folder. Is there a way to add that folder to /etc/pacman.conf so I can just do pacman -S <packagename>?
I tried adding:
[aur]
Server = file:///home/chris/aur/packages
to pacman.conf, but it spit out this error:
failed copying /home/chris/aur/packages/aur.db.tar.gz

failed to synchronize aur
error: could not open sync database: aur
have you used --refresh yet?

Am I just being stupid, or is it not possible to do that?

Snowman · 2005-06-23 03:01:03

It's not possible. The above script only fetch PKGBUILDs. You need to make the packages with makepkg and install them with pacman -A

Take a look at srcpac. IIRC, it makes the package and install it in one step.

lazychris2000 · 2005-06-23 03:15:51

Thanks. I'll give that a shot.

T-Dawg · 2005-06-23 09:29:56

you'll have to move them into /var/abs/local. srcpac does a top level find in /var/abs....

Arch Linux

#1 2005-04-29 19:31:43

web scraping + AUR = fun

#2 2005-04-29 21:14:51

Re: web scraping + AUR = fun

#3 2005-04-29 21:19:38

Re: web scraping + AUR = fun

#4 2005-04-29 22:08:27

Re: web scraping + AUR = fun

#5 2005-06-23 02:50:26

Re: web scraping + AUR = fun

#6 2005-06-23 03:01:03

Re: web scraping + AUR = fun

#7 2005-06-23 03:15:51

Re: web scraping + AUR = fun

#8 2005-06-23 09:29:56

Re: web scraping + AUR = fun

Board footer