Arch Linux

fiod

Member

Registered: 2007-04-02

Posts: 205

How to "harvest" urls?

Hey,

Suppose I want to download some file, using aria2.
Lets say the archlinux iso .

Is there a way to automatically collect all the url's from some download page?
Say, from the archlinux download page - to collect all the urls of the iso (so I could pass
them to aria2, and not have to do that manually).

Thanks a lot
Fiod

finferflu

Forum Fellow

From: Manchester, UK

Registered: 2007-06-21

Posts: 1,899

Website

Re: How to "harvest" urls?

Try with:

elinks -dump http://www.archlinux.org/download/

And see if that is what you're looking for.
Obviously you need to install elinks first

---

Have you Syued today?
Free music for free people! | Earthlings

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." -- A. de Saint-Exupery

carlocci

Member

From: Padova - Italy

Registered: 2008-02-12

Posts: 368

Re: How to "harvest" urls?

wget http://www.archlinux.org/download/ -q -O- | awk 'BEGIN{ RS="href=\""; FS="\""} {print $1}'

maybe check the output for links without absolute paths first

Last edited by carlocci (2008-03-31 20:37:38)