Combined repo and AUR websearch

lenni · 2007-11-17 15:01:52

Hi,

To scratch my personal little itch (since I often first check the repo search, and then the AUR website) I have programmed a PHP script that webscrapes both the repo and AUR. It's not an amazing piece of technology and programmes like yaourt have been doing this for a long time.

Nevertheless, try it out on:

http://www.stgabrielshalls.org.uk/lenni/

Any feedback, feature request or similiar, please post in this thread.

wain · 2007-11-17 16:10:37

Very nice
Can you add Unsupported or Community label for packages from AUR ?

Sigi · 2007-11-17 17:49:19

Cool 8)

mivo · 2007-11-17 17:52:42

Thanks! I had been looking for something like this. Very useful.

gummibaerchen · 2007-11-17 17:57:49

You made it (into my Epiphany's search bookmarks)!

Really much better than first search al.org and then the AUR

Thanks!!

(Maybe something like this could be provided by Arch itself? That would make things much faster I think)

stonecrest · 2007-11-17 19:37:13

If I search for 'sonata', as a test, it only shows sonata-svn from unsupported and not sonata from extra.

MrWeatherbee · 2007-11-17 20:16:06

stonecrest wrote:

If I search for 'sonata', as a test, it only shows sonata-svn from unsupported and not sonata from extra.

Yea. I like the idea, the design, the interface and the implementation for the most part , but perhaps a slight tweak in the search algorithm is in order.

As another example of erroneous results: if you search 'prozilla', you get zero hits; if you search for 'download', 'prozilla' appears in the results.

I've bookmarked it anyway. Thanks.

lenni · 2007-11-17 20:25:10

I just realised, if the repo search returns only one hit you'll get send straight to the package page, rather than the results page, which kinda screws up the webscraping loop. I'll fix that and let you know.

Thanks for the comments.

miggols99 · 2007-11-17 21:00:21

I really like that. Good work.

lenni · 2007-11-22 20:06:38

If I search for 'sonata', as a test, it only shows sonata-svn from unsupported and not sonata from extra.

I have fixed this issue now. Any feedback is very welcome.

bluePoet · 2007-11-22 20:32:26

lenni wrote:

If I search for 'sonata', as a test, it only shows sonata-svn from unsupported and not sonata from extra.
I have fixed this issue now. Any feedback is very welcome.

Their both tagged [AUR] now.

MrWeatherbee · 2007-11-22 20:37:26

lenni wrote:

If I search for 'sonata', as a test, it only shows sonata-svn from unsupported and not sonata from extra.
I have fixed this issue now. Any feedback is very welcome.

I think you may have fixed it too much. Seems it wants to find 'sonata' when it shouldn't. For example, if I search 'notinrepo', I get a result:

 Results for "notinrepo"
Displaying hits 1 - 1

[AUR] Search Criteria
Keywords

'Search' is a hyperlink to http://archlinux.org/packages/search/?r … l&q=sonata

Another example: search for 'hello' yields this:

 Results for "hello"
Displaying hits 1 - 3

[AUR] Search Criteria
Keywords

[AUR] quarry 0.2.0-1
Quarry is a multi-purpose GUI for several board games, at present Go, Amazons and Othello.

[AUR] othello 1.9.4-1
Reversi game implementation with very strong AI

Again, the 'Search Criteria' result appears and is linked to Sonata.

The 'Search Criteria' hit comes up quite a bit and is always linked to Sonata, but I haven't come up with a consistent pattern. Maybe it's a feature I don't understand.

On a positive note, searching for 'prozilla' now works.

lenni · 2007-11-22 23:26:18

I'm looking into this. The implementation was obviously not as good as I thought.

Last edited by lenni (2007-11-23 20:56:00)

smurnjiff · 2007-11-22 23:36:02

Have you tried "psearch"?

cactus · 2007-11-23 00:12:05

fyi. for the aur, there is a json interface that is hopefully going to be rolled out soon.
I am considering a similar api for the main site.

In the meantime, you can find some info about it here: http://archlinux.org/pipermail/tur-user … 06132.html

and some examples of usage here: http://archlinux.org/pipermail/tur-user … 06139.html

MrWeatherbee · 2007-11-23 03:42:57

lenni wrote:

I'm looking into this. The implementation was obviously not so good as I thought.

The issue I described above as well as the one described by bluePoet are now fixed.

lenni · 2007-11-23 10:55:04

Yeah, I fixed that. There is only one issue where the repo search returns only one hit, as described by smurnjiff, but I'll sort that out tonight.

Last edited by lenni (2007-11-23 21:26:47)

lenni · 2007-11-23 21:26:55

I have fixed (I think) the remaining issues and added an opensearch plugin. Thanks all for your comments.

MrWeatherbee · 2007-11-24 04:33:35

lenni wrote:

I have fixed (I think) the remaining issues and added an opensearch plugin. Thanks all for your comments.

You're welcome. Thank you for your efforts as well.

By the way, if you are interested in another challenge and one that I believe fits neatly in with what you have already done, maybe you could add code (and a second input box) for searching through package file-listings. A recent discussion took place here regarding the subject:

http://bbs.archlinux.org/viewtopic.php?id=39468

The Google search string method works well enough in most cases:

archlinux.org inurl:packages/files name_of_file

but it does fail to produce results more than one would like in order to place full confidence in it.

Just a thought. Anyway, thanks again.

lenni · 2007-12-25 23:59:48

I finally got round to adding the community/unsupported label to the AUR results.

Enjoy!

faemir · 2007-12-28 00:14:39

Would it be possible to use it as a web shortcut in konqueror? How would I go about doing this?

lenni · 2007-12-28 09:57:21

I'm not sure about Konquerer, but Firefox supports opensearch plugins (the search box next to the location bar). I have added one to the website and it's advertised to the browser.

I googled around and someone said Konquerer doesn't support opensearch plugins (please correct me if I'm wrong). I don't use Konquerer and don't know what a web shortcut is but I assume you will need to use the following url:

http://www.stgabrielshalls.org.uk/lenni … yourpacket

Just replace yourpacket with what you would like to search for.

Stefan Husmann · 2009-01-20 18:05:22

is this page still working for others? For me it is not (conkeror, MS Internet Explorer 6, pleas do not hit me:)) because it does not show up the package I searched for but many others.

b9anders · 2009-01-20 19:58:34

Stefan Husmann wrote:

is this page still working for others? For me it is not (conkeror, MS Internet Explorer 6, pleas do not hit me:)) because it does not show up the package I searched for but many others.

I won't, but it has made me think about coding something that will allow to hit people across the web.:mad::P

Arch Linux

#1 2007-11-17 15:01:52

Combined repo and AUR websearch

#2 2007-11-17 16:10:37

Re: Combined repo and AUR websearch

#3 2007-11-17 17:49:19

Re: Combined repo and AUR websearch

#4 2007-11-17 17:52:42

Re: Combined repo and AUR websearch

#5 2007-11-17 17:57:49

Re: Combined repo and AUR websearch

#6 2007-11-17 19:37:13

Re: Combined repo and AUR websearch

#7 2007-11-17 20:16:06

Re: Combined repo and AUR websearch

#8 2007-11-17 20:25:10

Re: Combined repo and AUR websearch

#9 2007-11-17 21:00:21

Re: Combined repo and AUR websearch

#10 2007-11-22 20:06:38

Re: Combined repo and AUR websearch

#11 2007-11-22 20:32:26

Re: Combined repo and AUR websearch

#12 2007-11-22 20:37:26

Re: Combined repo and AUR websearch

#13 2007-11-22 23:26:18

Re: Combined repo and AUR websearch

#14 2007-11-22 23:36:02

Re: Combined repo and AUR websearch

#15 2007-11-23 00:12:05

Re: Combined repo and AUR websearch

#16 2007-11-23 03:42:57

Re: Combined repo and AUR websearch

#17 2007-11-23 10:55:04

Re: Combined repo and AUR websearch

#18 2007-11-23 21:26:55

Re: Combined repo and AUR websearch

#19 2007-11-24 04:33:35

Re: Combined repo and AUR websearch

#20 2007-12-25 23:59:48

Re: Combined repo and AUR websearch

#21 2007-12-28 00:14:39

Re: Combined repo and AUR websearch

#22 2007-12-28 09:57:21

Re: Combined repo and AUR websearch

#23 2009-01-20 18:05:22

Re: Combined repo and AUR websearch

#24 2009-01-20 19:58:34

Re: Combined repo and AUR websearch

Board footer