Arch Linux

Davigetto

Member

From: In your mind

Registered: 2007-05-10

Posts: 266

A shellscript for saving web images from a website. Is it possible?

Hello, I am a designer, and well, I often query the following website to look for some images:

http://www.deviantart.com/

As you can see, there are some images. These images changes over the time.

Can be done a shellscript to query this website (only the main page) for example every hour and automatically download the new images?

It can be a easy work or a really hard task?

Greetings

---

Only deaths can see the end of battles.

Blog: http://djmartinez.co.cc -> The life of a Computer Engineer

gunnihinn

Member

From: Torreón, Mexico

Registered: 2007-10-28

Posts: 81

Re: A shellscript for saving web images from a website. Is it possible?

Beware; no real useful advice follows.

If it can be done, and off the top of my head that depends a bit on how deviantart files its new addings, it's probably better done with Perl, Python or something like that than a shell script.

Shell scripts are the devils own work, and should be resorted to only for dirty-drunk-last-minute-hacks.

elide

Member

From: Russia

Registered: 2007-12-02

Posts: 40

Re: A shellscript for saving web images from a website. Is it possible?

#!/bin/bash

wget -U 'SomeUserAgent/1.0' -O- 'http://www.deviantart.com/' 2> /dev/null |
grep -Po 'http://[^.]+\.deviantart.com/art/[^"]+-\d+' |
sed -r 's/.+-([0-9]+)/http:\/\/www.deviantart.com\/download\/\1\/xx/' |
wget -U 'SomeUserAgent/1.0' -i-

try this.

Fackamato

Member

Registered: 2006-03-31

Posts: 579

Re: A shellscript for saving web images from a website. Is it possible?

#!/bin/bash

wget -U 'SomeUserAgent/1.0' -O- 'http://www.deviantart.com/' 2> /dev/null |
grep -Po 'http://[^.]+\.deviantart.com/art/[^"]+-\d+' |
sed -r 's/.+-([0-9]+)/http:\/\/www.deviantart.com\/download\/\1\/xx/' |
wget -U 'SomeUserAgent/1.0' -i-

try this.

Works great! I wonder how nice this is to deviantart though..

Cerebral

Forum Fellow

From: Waterloo, ON, CA

Registered: 2005-04-08

Posts: 3,108

Website

Re: A shellscript for saving web images from a website. Is it possible?

Shell scripts are the devils own work, and should be resorted to only for dirty-drunk-last-minute-hacks.

I don't follow this logic - shell scripts are great.  Using a shell script, elide got the images from the front page in 4 lines.  Try doing that with perl or python.

sime

Member

Registered: 2007-12-14

Posts: 96

Re: A shellscript for saving web images from a website. Is it possible?

I don't follow this logic - shell scripts are great.  Using a shell script, elide got the images from the front page in 4 lines.  Try doing that with perl or python.

Well, to be fair, with Perl, you could actually achieve the same in less then the example shell script provided here.

Last edited by sime (2007-12-18 21:39:27)

elide

Member

From: Russia

Registered: 2007-12-02

Posts: 40

Re: A shellscript for saving web images from a website. Is it possible?

with Perl, you could actually achieve the same in less

show me the code (;

Last edited by elide (2007-12-18 22:14:05)

sime

Member

Registered: 2007-12-14

Posts: 96

Re: A shellscript for saving web images from a website. Is it possible?

show me the code (;

Show me the moneeey!!1 This isn't a pissing contest, I just pointed out the incorrect fact.

mezoko

Member

Registered: 2005-03-26

Posts: 310

Website

Re: A shellscript for saving web images from a website. Is it possible?

show me the code (;

Show me the moneeey!!1 This isn't a pissing contest, I just pointed out the incorrect fact.

Ok so then backup your statement! In addition the bash method took maybe 5 minutes it seems if you want money for you solution it'd take much much longer!

---

"The only thing we have to fear is fear itself." - Franklin D. Roosevelt

dumas

Member

From: Sydney

Registered: 2007-09-01

Posts: 103

Re: A shellscript for saving web images from a website. Is it possible?

By the way, I wrote a bash script to trim long lines of text file into lines 70 characters. It takes 100% CPU performance and outputs nearly a line per second. Is this just crappy scripting from me or bottleneck from bash?

#!/bin/bash
while read line
do
    while [ -n "$line" ]
    do
        word=`echo "$line" | cut -d' ' -f1`
        if [ -n "$newline ]
        then
            newline="$newline $word"
        else
            newline=$word
        fi
        if [ "$word" == "$line" ]
        then
           line=''
        else
            line=`echo "$line" | cut -d' ' -f1 --complement`
        fi
        len=`echo "$newline" | wc -m`
        if [ $len -ge 70 ]
        then
            echo "$newline"
            newline=''
        fi
    done
    echo "$newline"
    newline=''
done

sime

Member

Registered: 2007-12-14

Posts: 96

Re: A shellscript for saving web images from a website. Is it possible?

In addition the bash method took maybe 5 minutes it seems if you want money for you solution it'd take much much longer!

It was a joke (Jerry Maguire)…

skymt

Member

Registered: 2006-11-27

Posts: 443

Re: A shellscript for saving web images from a website. Is it possible?

By the way, I wrote a bash script to trim long lines of text file into lines 70 characters. It takes 100% CPU performance and outputs nearly a line per second. Is this just crappy scripting from me or bottleneck from bash?

<code snipped>

Either way, it would probably be better to just use the fold command.