Arch Linux

zcyrus

Member

Registered: 2009-11-02

Posts: 9

What console tool can I use to parse a website for custom links?

For example, I dont want to use a browser to visit a tracker every time and need to get a list of .torrent files at it for aria2c. Result must a be a list of absolute urls seperated with spaces. It must need just a url to grab everything starting from it recursively. Struggling with it for an enormous amount of time already :\ !

Last edited by zcyrus (2010-06-06 18:07:28)

awkwood

Member

From: .au <=> .ca

Registered: 2009-04-23

Posts: 91

Re: What console tool can I use to parse a website for custom links?

If you have ruby installed:

curl -s http://your-torrent-site.com | ruby torrents.rb

torrents.rb

#!/usr/bin/env ruby
# extracts torrent links from STDIN
require 'uri'
protocols = %w[http]

while line = gets
  protocols.map{ |pr| URI::extract(line, pr) }.
    flatten.compact.delete_if{ |uri| uri !~ /\.torrent\Z/i }.
    each{ |uri| print uri + ' ' }
end

brisbin33

Member

From: boston, ma

Registered: 2008-07-24

Posts: 1,799

Website

Re: What console tool can I use to parse a website for custom links?

in this specific scenario i like to use lynx -dump because it'll put all the embedded links numbered at the bottom for easy regex'ing. 

i use this method for parsing thepiratebay.org search results:

lynx -dump www.foo.com | sed -e '/^[^ ]* .*\/\([^\/]*\)\.\([0-9]*\)\.TPB.torrent$/!d;s//\2|\1/g'

in my case, i use the back references to get a "$name|$id" out of each link but you get the idea.

---

//github/

Mr.Elendig

#archlinux@freenode channel op

From: The intertubes

Registered: 2004-11-07

Posts: 4,097

Re: What console tool can I use to parse a website for custom links?

Imo the best way is to use (python with lxml.html) or similar. Parsing html with regexp really hard to get right, and prone to errors.

(specially since people often writes malformed html)

Last edited by Mr.Elendig (2010-06-06 20:31:04)

---

Evil #archlinux@libera.chat channel op and general support dude.
. files on github, Screenshots, Random pics and the rest

dmz

Member

From: Sweden

Registered: 2008-08-27

Posts: 881

Website

Re: What console tool can I use to parse a website for custom links?

http://search.cpan.org/dist/WWW-Mechani … chanize.pm is amazing.

---

ls++ | LS_COLORS project | file management in vim | tons of utilities | configs | japh.se

awkwood

Member

From: .au <=> .ca

Registered: 2009-04-23

Posts: 91

Re: What console tool can I use to parse a website for custom links?

Here's the original file in case you want to fetch all of the links from a page instead of just torrents. This outputs one link per line.

uris.rb

 
#!/usr/bin/env ruby
# extracts all links from STDIN
require 'uri'
protocols = %w[http ftp]

while line = gets
  protocols.map{ |pr| URI::extract(line, pr) }.
    flatten.compact.each{ |uri| puts uri }
end