Script : checks if webpage changed with login:pass. Finished!

Ventil1 · 2010-03-19 17:32:03

I need a .sh script to monitor a webpage and then notify me in conky if there is something new on the site. The site I need to monitor uses login and pass.
I know that there are great applications like specto for this. But I just want a simle .sh script. Not python! Don't want to use python!

I've found 1 but can't figure it out. This is all new to me. Plus it doesn't use login:pass. And it doesn't work for me

#!/usr/bin/env bash
url = "http://www.google.com"
checkfile = "~/pagecheck.md5"
email  = "you@domain.tld"
savefile = "/dev/null"
[ -f ./pagecheck ] \
  && \
  (diff $(wget -O $savefile $url | md5sum) $checkfile) && mail -s "UPDATED: $url" $email) \
  || \
  (wget -O $savefile $url | md5sum > $checkfile)

Does anyone know a good script I could use?

Last edited by Ventil1 (2010-03-21 22:26:09)

brisbin33 · 2010-03-19 17:56:28

maybe if i rewrite the script a little bit it'll make more sense:

#!/bin/bash

# your email address
email='you@domain.tld'

# site to check
url='http://www.google.com'

# temp file
checkfile="$HOME/.pagecheck.md5"

if [ -f "$checkfile" ]; then

  md5_here="$(cat "$checkfile")"
  md5_there="$(wget -O - "$url" | md5sum)"

  if [ "$md5_here" != "$md5_there" ]; then
    # site's changed
    mail -s "UPDATED: $url" $email
  fi
else
  # first run
  wget -O - "$url" | md5sum > "$checkfile"
fi

now if you follow that, just man wget for the user and password options. if those don't work, lynx has similar options and --dump will put the page on stdout to be passed to md5sum.

note: untested.

/me was bored...

Pierre · 2010-03-19 18:22:20

I have written a simple script for this: w3watch (available in AUR)

falconindy · 2010-03-19 18:30:59

does wget support anything other than HTTP based authentication? If the login/pass are sent in a form with POST (or GET, eek), i think you'll need to use curl to authenticate, set a cookie, and grab the page.

Ventil1 · 2010-03-20 08:39:56

Thanks for the help guys. But like I said... This is new to me. I am not a programer. I don't even understand half the stuff you guys said.

I need it similar to this. I mean the if else part.

checkgmail.sh | How to make conky+checkgmail.sh to play .wav file? [Solved]

#!/bin/bash

gmail_login="myusername"  
gmail_password="mypass"

nbmail="$(wget --secure-protocol=TLSv1 --timeout=3 -t 1 -q -O - \
https://${gmail_login}:${gmail_password}@mail.google.com/mail/feed/atom \
--no-check-certificate | grep 'fullcount' \
| sed -e 's/.*<fullcount>//;s/<\/fullcount>.*//' 2>/dev/null)"

if [ -z "$nbmail" ]; then
echo "Error"
else
  if [ $nbmail != 0 ]; then
    echo "$nbmail message(s)"
    aplay ~/sounds/mail_here.wav
  else
  echo "$nbmail message(s)"
  fi
fi

Pierre where do I put the login and pass?

# Maintainer: Pierre Schmitz <pierre@archlinux.de>

pkgname=w3watch-git
pkgver=20090827
pkgrel=1
pkgdesc='Watch the web and get notified on updates'
arch=('any')
url='http://git.archlinux.de/w3watch/'
depends=('bash' 'lynx' 'awk' 'diffutils' 'coreutils' 'grep')
license=('GPL')

_gitroot='http://git.archlinux.de/~pierre/w3watch.git'
_gitname='w3watch'

build() {
    cd ${srcdir}
    msg 'Connecting to GIT server....'

    if [ -d $_gitname ] ; then
        pushd $_gitname && git pull origin
        msg 'The local files are updated.'
        popd
    else
        git clone $_gitroot
    fi

    msg 'GIT checkout done or server timeout'
    msg 'Starting make...'
    install -D -m755 "${_gitname}/w3watch" \
        "${pkgdir}/usr/bin/w3watch"
    install -D -m644 "${_gitname}/config.sample" \
        "${pkgdir}/usr/share/doc/w3watch/config.sample"
}

Last edited by Ventil1 (2010-03-20 08:55:31)

Ventil1 · 2010-03-20 08:47:40

brisbin33 wrote:

maybe if i rewrite the script a little bit it'll make more sense:
note: untested.
/me was bored...

This works brisbin33! But I am not sure what it does?

This was the output:

[marko@Asus ~]$ sh nova.sh 
--2010-03-20 09:42:10--  http://www.google.com/
Razrešuje se www.google.com...74.125.43.104, 74.125.43.105, 74.125.43.106, ...
Povezujem se na www.google.com|74.125.43.104|:80... priključen.
HTTP zahteva poslana, čakam odgovor... 302 Found
Položaj: http://www.google.si/ [spremljam]
--2010-03-20 09:42:10--  http://www.google.si/
Razrešuje se www.google.si...74.125.43.105, 74.125.43.106, 74.125.43.147, ...
Znova se uporablja povezava z www.google.com:80.
HTTP zahteva poslana, čakam odgovor... 200 OK
Dolžina: nedoločen [text/html]
Saving to: `STDOUT'

    [ <=>                                   ] 7244        --.-K/s   v 0,07s    

2010-03-20 09:42:10 (101 KB/s) - written to stdout [7244]

Can you please get bored some more and then finish the script so I can put in:

site_login="myusername"  
sitel_password="mypass"

And it does this:

else
  if [ $nbmail != 0 ]; then
    echo "site changed"
    aplay ~/sounds/sound_file.wav
  else
  echo "site hasn't changed"
  fi
fi

Last edited by Ventil1 (2010-03-20 08:50:12)

Ventil1 · 2010-03-20 09:08:22

Its almost there:

#!/bin/bash


# site to check
url='http://www.google.com'
website_login='myusername' 
websitel_password='mypass' 

# temp file
checkfile="$HOME/.pagecheck.md5"

if [ -f "$checkfile" ]; then

  md5_here="$(cat "$checkfile")"
  md5_there="$(wget -O - --user='$website_login' --password='$website_password'  "$url" | md5sum)"

  if [ "$md5_here" != "$md5_there" ]; then
    # site's changed
    echo "site changed"
    aplay ~/sounds/sound_file.wav
  else
  echo "site hasn't changed"
  fi
else
  # first run
  wget -O - --user='$website_login' --password='$website_password'   "$url" | md5sum > "$checkfile"
fi

Something is not working right, yet. It tells me that the site changed every time I visit it. I think I get different md5sum each time

Last edited by Ventil1 (2010-03-20 10:03:22)

brisbin33 · 2010-03-20 14:57:08

by using '$website_login' you're attempting to authenticate with literally '$website-login'. use double quotes to expand variables. see if that fixes it.

/edit: also, try wget -q

Last edited by brisbin33 (2010-03-20 14:57:44)

Ventil1 · 2010-03-20 16:40:11

I am affraid it is a little more complicated. I can't not even login to the site.

This is the site's login.php file I need to login:

<form id="login" method="POST" action="/user/login.php"> 
<input type="hidden" value="" name="previous"> 
<label>Username</label> 
<br/> 
<input name="uname" type="text" class="textinput" /> 
<br/> 
<br/> 
<label>Password</label> 
<br/> 
<input name="pword" type="password" class="textinput" /> 
<br/> 
<input type="image" src="/images/login/login.gif" width="120" height="40" vspace="10" /><br /> 
<label> 
  <a href="/tips/logfail.php">Login Problems</a> 
</label> 
</form>

#wget -Opage1.html  --cookies=on --save-cookies=cookies.txt --keep-session-cookies --post-data "previous=''&uname=MYUSER&pword=MYPASS" http://www.page1/login.php

#wget -Opage2.html --cookies=on --keep-session-cookies --load-cookies=cookies.txt "http://www.page1/page2"

The downloaded page2.html file is the same as if I wasn't logged in.

This is the cookie.txt file:

# HTTP cookie file.
# Generated by Wget on 2010-03-20 17:22:44.
# Edit at your own risk.

www.page1.com       FALSE    /    FALSE    0    PHPSESSID    4f34265d31ada7db7d141102af05913d

Last edited by Ventil1 (2010-03-21 11:55:38)

Ventil1 · 2010-03-21 11:37:29

It's working!! woohoo! I just need help with a few small fixes.
1. I need to copy file1.html file to file2.html (see the code where ####)
2. I don't think it reads the file1.html and file2.html well. When it compares them it allways say the site changed.
3. I want the script to tolerate comparisment by %. (If file1.html is 98% similar to file2.html then the site hasn't changed)
4. I want output to be just echo "Site changed" or "Site hasn't changed"

#!/bin/bash


#site login
url='http://www.site.com/'

#username, password
user='MYUSERNAME'  
pass='MYPASS'


# temp file
file1="$HOME/file1.html"
file2="$HOME/file2.html"

if [ -f "$file1" ]; then
  wget  -O - --save-cookies=cookies.txt  --keep-session-cookies \
  --post-data "uname=$user&pword=$pass" ${url}login.php
  wget -O "$file2" --keep-session-cookies --load-cookies=cookies.txt \
    "${url}file.pdf"
  check_here="$(cat "$file1")"
  check_there="$(cat "$file2")"

  if [ "$check_here" != "$check_there" ]; then
    # site's changed
    echo "site changed"
    aplay ~/Downloads/mail_here.wav
######    This is where I am stuck!! I need to copy file "file2.html" to "file1.html"  ######
  else
  echo "site hasn't changed"
  fi
else
  # first run
  wget  -O - --save-cookies=cookies.txt  --keep-session-cookies \
  --post-data "uname=$user&pword=$pass" ${url}login.php
  wget -O "$file1" --keep-session-cookies --load-cookies=cookies.txt \
  "${url}file.pdf"
fi

Last edited by Ventil1 (2010-03-21 15:34:04)

Ventil1 · 2010-03-21 22:24:41

I've done it! I've done it!! MY 1st script!

#!/bin/bash


#site login
url='http://www.site.com'

#username, password
user='MyUSERNAME'  
pass='MyPASSWORD'

#check if file.html file does exist
if [ -f file.html ]; then
   # temp file size
   OLDFILE_SIZE=`cat file.html`

   wget  -O - --save-cookies=cookies.txt --keep-session-cookies --post-data "uname=$user&pword=$pass" ${url}/login.php
   HTML_STRING="$(wget -O - --keep-session-cookies --load-cookies=cookies.txt \
    "${url}/members/stuff.html")"
   NEWFILE_SIZE=${#HTML_STRING}

   if [[ $((${#OLDFILE_SIZE})) -gt $(($NEWFILE_SIZE-10)) && $((${#OLDFILE_SIZE})) -lt $(($NEWFILE_SIZE+10)) ]]
   then
      exit 1;
   else
      dialog  --title "site changed" --msgbox "Site Changed!" 10 30
      echo -e "$HTML_STRING" > file.html
   fi
else
   # run if file.html does not exist yet
   wget  -O - --save-cookies=cookies.txt --keep-session-cookies --post-data "uname=$user&pword=$pass" ${url}/login.php
   wget -O file.html --keep-session-cookies --load-cookies=cookies.txt \
    "${url}/members/stuff.html"
fi

Arch Linux

#1 2010-03-19 17:32:03

Script : checks if webpage changed with login:pass. Finished!

#2 2010-03-19 17:56:28

Re: Script : checks if webpage changed with login:pass. Finished!

#3 2010-03-19 18:22:20

Re: Script : checks if webpage changed with login:pass. Finished!

#4 2010-03-19 18:30:59

Re: Script : checks if webpage changed with login:pass. Finished!

#5 2010-03-20 08:39:56

Re: Script : checks if webpage changed with login:pass. Finished!

#6 2010-03-20 08:47:40

Re: Script : checks if webpage changed with login:pass. Finished!

#7 2010-03-20 09:08:22

Re: Script : checks if webpage changed with login:pass. Finished!

#8 2010-03-20 14:57:08

Re: Script : checks if webpage changed with login:pass. Finished!

#9 2010-03-20 16:40:11

Re: Script : checks if webpage changed with login:pass. Finished!

#10 2010-03-21 11:37:29

Re: Script : checks if webpage changed with login:pass. Finished!

#11 2010-03-21 22:24:41

Re: Script : checks if webpage changed with login:pass. Finished!

Board footer