Arch Linux

lenni

Member

From: London, UK

Registered: 2007-09-21

Posts: 95

AUR API or web scraping instructions?

I'm trying to scratch my personal pet itch and also teach myself PHP: I would like to program a meta-search engine that searches both the repo and AUR.

To web scrape the repo is easy 'cos its an GET HTTP request, however AUR uses POST (for some reason). Can anyone tell me which headers I need to put into my POST request in order to get the AUR response.

So far I have:

'L'='0'
'C'='0'
'SeB'='nd'
'Pas'='all'
'SB'='n'
'SO'='a'
'PP'='100'

'K'=$thesearchstring

Is that it?

Thanks.

---

Combined repo and AUR search

tardo

Member

Registered: 2006-07-15

Posts: 526

Re: AUR API or web scraping instructions?

PP = packages per page. Accepted values: 25, 50, 75, 100. Default: 25.
L   = location ID. Accepted values: 2, 3. 2 = unsupported, 3 = community.
C  = Category ID. Accepted values: 1-18: in numerical order: none, daemons, devel, editors, emulators, games, gnome, il8n, kde, lib, modules, multimedia, network, office, science, system, x11, xfce.
SeB = Search by: Accepted values: m, s: m = maintainer name, s = user id (numerical val)
PaS = search for safe/unsafe/all packages. Accepted values: all, fs, us. Should be self-explanatory.
SO = sort order. Accepted values: a, d. ascending/descending.
SB = sort by. Accepted values: c,l,v,m,a. In order, category, location, votes, username, age
K = search string.

other vars:
do_Search = perform a search. set it to 1.
do_MyPackages = self-explanatory. set to 1.
do_Orphans = also self-explanatory. set to 1.

MyPackages required you to be logged in, dunno if that's possible just scraping.

byte

Member

From: Düsseldorf (DE)

Registered: 2006-05-01

Posts: 2,046

Re: AUR API or web scraping instructions?

> To web scrape the repo is easy 'cos its an GET HTTP request, however AUR uses POST (for some reason).

How does that matter? You don't have to use POST if you don't want to. Tack your parameters to packages.php and off you go.

---

1000

lenni

Member

From: London, UK

Registered: 2007-09-21

Posts: 95

Re: AUR API or web scraping instructions?

Okay, I'll give that a try.

I always thought, that POST data has to be send in the header and not in the URL.

Thanks.

---

Combined repo and AUR search

byte

Member

From: Düsseldorf (DE)

Registered: 2006-05-01

Posts: 2,046

Re: AUR API or web scraping instructions?

Well, if it's in the URL, it's GET. Choose whichever you want. Most of the time the server doesn't enforce one method of form input. And GET is generally the preferred method for queries which don't manipulate server data.

---

1000

neotuli

Lazy Developer

From: London, UK

Registered: 2004-07-06

Posts: 1,204

Website

Re: AUR API or web scraping instructions?

http://archlinux.org/pipermail/tur-user … 06131.html

---

The suggestion box only accepts patches.