[solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

whoops · 2014-02-23 16:37:09

Hello!

I need to wget a few (maybe 20 -.- ) html files that are linked on one html page (same domain) recursively, but the robots.txt there disallows wget. Now I could just ignore the robots.txt... but then my wget would also ignore the info on forbidden links to dynamic sites which are forbidden in the very same robots.txt for good reasons. And I don't want my wget pressing random buttons on that site. Which is what the robots.txt is for. But I can't use the robots.txt with wget.

Any hints on how to do this (with wget)?

Last edited by whoops (2014-02-23 17:52:31)

Spider.007 · 2014-02-23 16:39:50

Would wget ignore that part of the robots.txt if you specify a different user-agent?

whoops · 2014-02-23 16:58:21

That was the first thing I tried, didn't seem to make a difference. Should it / did I do it wrong?

HalosGhost · 2014-02-23 17:13:19

I know suggesting an alternative is not really a solution. But curl is significantly faster, more powerful and provides a native C interface. Have you tried using it? Or, is there a specific reason you must use wget?

All the best,

-HG

Last edited by HalosGhost (2014-02-23 17:13:57)

Spider.007 · 2014-02-23 17:20:03

whoops wrote:

That was the first thing I tried, didn't seem to make a difference. Should it / did I do it wrong?

No it was a wild guess

whoops · 2014-02-23 17:51:01

HalosGhost wrote:

Have you tried using it? Or, is there a specific reason you must use wget?

Only stubborness
Stupid website -.- what do they even think they achieve by disallowing wget? I should just use the ignore option and let wget "click" on every single button in their php interface. But nooo, instead I waste time trying to figure out a way to exclude those GUI links from being followed even though wget would be perfectly set up to comply to that automatically if it weren't for that one entry to "ban" it. *grml*

Will definitely try curl next time though - thanks for the suggestion!

And now, I present...
THE ULTIMATIVE SOLUTION**:

sudo sed -i 's/wget/wgot/' /usr/bin/wget

YAY.

./solved!

** stubborn version.

Last edited by whoops (2014-02-23 17:51:19)

Spider.007 · 2014-02-24 13:31:15

That is... crafty

Arch Linux

#1 2014-02-23 16:37:09

[solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

#2 2014-02-23 16:39:50

Re: [solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

#3 2014-02-23 16:58:21

Re: [solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

#4 2014-02-23 17:13:19

Re: [solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

#5 2014-02-23 17:20:03

Re: [solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

#6 2014-02-23 17:51:01

Re: [solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

#7 2014-02-24 13:31:15

Re: [solved]Wget: ignore "disallow wget" +comply to the rest of robots.txt

Board footer