Arch Linux

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

[Solved] Script to monitor log file, and send email if certain...

Hi all,

Let me first of all explain what the script must do and the complication I am having.

- The script will monitor the access.log file for certain words, once those word are found within the log, the script will then email the line (address visited) where the word was found.

I am expected to found certain word quite often while the user will browse, therefore the script should run as a loop and check the log every given minutes. Also the log will become rather large therefore if possible the script should the last 5 minutes of log (in order to limit the resource required and the speed of action).

This is too advance for me, so I would appreciate some guidance should you wish to assist,

Looking forward for your replies,

Last edited by sweetthdevil (2011-11-24 15:09:09)

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [Solved] Script to monitor log file, and send email if certain...

grep word /path/to/access.log | mailx -s subject address@host

Use a cronjob or 'watch' to run periodically.
This will return all the lines from access.log that have 'word' in them. You need to rotate the log before you run grep or you will be getting the old hits over and over (+ the new ones):
Let's say, you're looking for the word 'cat'. After the first run, two cats are found

cat 1
cat 2

after the next run, grep found two more

cat 1
cat 2
cat 3
cat 4

and in the next run, additional two

cat 1
cat 2
cat 3
cat 4
cat 5
cat 6

If you'd prefer to get

cat 1
cat 2

then

cat 3
cat 4

followed by

cat 5
cat 6

you need to figure it out yourself ;P

Last edited by karol (2011-11-22 16:54:53)

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

Re: [Solved] Script to monitor log file, and send email if certain...

Hi,

First of all, man thanks for your reply.

As you can see, I seems to have an issue with mailx. Also I would need some how to translate the timestamps to a human readable time for the email.

And how would you return the log before (access.log is own by proxy) other user only have read-only (don't want to change it)

[sweetth@myhost ~]$ grep word /var/log/squid/access.log | mailx -s Alert sweetthdevil@****.***
/usr/sbin/sendmail: No such file or directory
[sweetth@myhost ~]$ "/home/sweetth/dead.letter" 70/9671
. . . message not sent.

Many thanks for your help,

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [Solved] Script to monitor log file, and send email if certain...

You can use some other tool to send messages, I use mailx + gmail account.
You have to configure mailx http://www.archlinux.org/packages/?name=mailx :

[karol@black ~]$ head /etc/mail.rc
set sendmail="/usr/bin/mailx"

set smtp=smtp.gmail.com:587
set smtp-use-starttls
set ssl-verify=ignore
set ssl-auth=login
set smtp-auth-user=your.login@gmail.com
set smtp-auth-password=yourpassword

I simply edited /etc/mail.rc and added the above lines to the top of the file. Remember to use your own login and password (edit the last two lines) but don't post them here :-)
The first line in the above config takes care of the '/usr/sbin/sendmail: No such file or directory' error.

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

Re: [Solved] Script to monitor log file, and send email if certain...

Great that work!

Now I need to figure out how to use the timestamps to show only the latest one and to made the timestamps readable on the email...

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [Solved] Script to monitor log file, and send email if certain...

Now I need to figure out how to use the timestamps to show only the latest one and to made the timestamps readable on the email...

You can try http://stackoverflow.com/questions/4331 … -a-command or create a temp file and diff (or 'comm -23') the previous results of the grep command with the current ones but if the log gets rotated often, this won't work well.

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

Re: [Solved] Script to monitor log file, and send email if certain...

Would it be possible to put the result of the grep command into a string, than compare the time stamps and remove the one older than x minutes?

Than it will email if the string isn't empty?

Regards,

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [Solved] Script to monitor log file, and send email if certain...

I though about using a temp file to collect what we've already found & send, let's call it 'cats'.

#!/bin/bash

#if there's no 'cats' file
if [ ! -e cats ] ; then
# let's search for the word 'cat' in our log and save the output to the newly created file named 'cats'
  grep cat access.log > cats
# and mail its contents
  mailx -s subject address@gmail.com < cats
# if however the 'cats' file already exists
else
# it should contain some cats already, so let's send just the new ones
  comm -23 <(grep cat access.log) cats | mailx -s subject address@gmail.com
# and update our cats log
  grep cat access.log > cats
fi

I created a file called 'access.log' in the same directory as this script and populated it with

cat 1
dog
cat 2

After I run the script, the 'cats' file got created with

cat 1
cat 2

and its contents have been mailed to me.
Now if I update the log to, say

cat 1
dog
cat 2
dog
cat 3
cat 4

and run the script again, the cats file will get modified to

cat 1
cat 2
cat 3
cat 4

and I will get

cat 3
cat 4

in the mail - I've already got 'cat 1' and 'cat 2' on the previous run.

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

Re: [Solved] Script to monitor log file, and send email if certain...

Dear karol,

Many thanks for your help, it work like a charm! The only change I must do (hopefully with your assistance one last time)

I will use cronjob to run the script on a regular basic, issue is that I will receive an email every time the script is run regardless if there is anything to report, therefore ideally the script would check whether or not there is content to send and act accordingly (send should there is, not sending should there isn't)

Again, many time for your assistance and time,

Regards,

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

Re: [Solved] Script to monitor log file, and send email if certain...

Right,

with mailx there is an option not to send an email with an empty message "mailx -E"

All sorted!!

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [Solved] Script to monitor log file, and send email if certain...

Yup, I think it's the easiest way to do it.
I was about to suggest this, but part of me wanted to write 'c'mon, you just have grep and mailx there, so reading 2 man pages shouldn't be that hard' and decided to wait a couple more hours ;P

Please remember to mark the thread as solved.

sweetthdevil

Member

Registered: 2009-10-20

Posts: 415

Re: [Solved] Script to monitor log file, and send email if certain...

No you are absolutely right, I was trying since this morning to parse the log into a string $hst_file and use and if to see if it wasn't empty....

Why making it difficult when linux is so simple ^^

Again many many thanks for your assistance and time!