Arch Linux

ScannerDarkly

Member

From: UK

Registered: 2010-07-24

Posts: 20

Website

Dir monitoring, Text manipulation

Scenario: rsync. The directory ~/logs/hostname/ contains lots of log files. These files are timestamped in a yyyymmdd format, and are created daily. They contain lots of information, being difficult to read the bits I need at a glance.

I would like some pointers how I can:

• check for newly created log files in that directory structure (~/logs/hostname1; ~/logs/hostname2 -- up to 5)

• copy the whole line containing keywords like "bytes, speed, etc"

• Take that whole line from the log, move it into a new file (if necessary?)

• Format it e.g. "Bytes transferred: ..." , "Speed of transfer: ...", etc

• use mailx to e-mail me, with an appropriate subject

The finished script offers a rudimentary alert system. This is my own little project, so I want to learn. What are the best tools to use here?

Thanks

Last edited by ScannerDarkly (2011-01-09 09:59:38)

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: Dir monitoring, Text manipulation

Here's a simple script https://bbs.archlinux.org/viewtopic.php … 15#p462415 that gathers some info and displays it in a certain manner. You too will need tools like grep and/or awk.

* check for newly created log files in that directory structure (~/logs/hostname1; ~/logs/hostname2 -- up to 5)

Use the filenames as arguments for the grep command, as I did with 'test1'.

* copy the whole line containing keywords like "bytes, speed, etc"
* Take that whole line from the log, move it into a new file (if necessary?)

That's what e.g. grep does. Imagine I have a file with many records (a fancy name for lines) and I want to copy only those that have 'a' somewhere in there:

[karol@black ~]$ cat test1
a
b
c
aa
ab
bc
[karol@black ~]$ grep a test1
a
aa
ab
[karol@black ~]$ grep a test1 > only_a
[karol@black ~]$ cat only_a 
a
aa
ab

'grep a test1' picks only lines with 'a'.
'grep a test1 > only_a' sends the output of 'grep a test1' to another file, which I named 'only_a'.

If it's all new to you, find some tutorials on grep + read the man page. Some bash scripting can also come in handy http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashFAQ

use mailx to e-mail me, with an appropriate subject

I use this for my mailx-heirloom + gmail account:

echo "Done." | mailx -s backup.weekly -a attachment.tgz some.login@gmail.com

ScannerDarkly

Member

From: UK

Registered: 2010-07-24

Posts: 20

Website

Re: Dir monitoring, Text manipulation

But how will I grep files when there are going to be new ones in the directory every day? I thought I could check the time they were made with find, or diff. Then I do not know how to act upon the files it finds..

I can cron this after the logfiles are made:
if file < 5hrs old
then
grep words...

How would I write something like this?

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: Dir monitoring, Text manipulation

Read the man page for 'find' - it can select files that are 'older then ...' etc.

find /path/to/files* -mtime -1 -exec grep <blah blah blah> {} \;

or you can use the filenames

find . -name "$(date +%Y%m%d)"

to pick just the file created today..

It's a bit ugly, but you can even

find . -name "$(( $(date +%Y%m%d) -1 ))"

to pick ones from yesterday etc.

If the files are like

20110108
20110109
20110110

it should be enough, but if the filenames are

foo-20110108.log
foo-20110109.log
foo-20110110.log

you need to tell 'find' to look for

find . -name "*$(date +%Y%m%d)*"

Last edited by karol (2011-01-09 17:51:50)

awkwood

Member

From: .au <=> .ca

Registered: 2009-04-23

Posts: 91

Re: Dir monitoring, Text manipulation

You could use a Ruby library called watchr to monitor the log directory in realtime.
It's build for continuous integration but it's quite flexible.

To install (you'll need Ruby first obviously) just type:

gem install watchr rev mail

Once these rubygems are installed create a file like the one below in ~/logs and run it like so:

watchr logs.watchr

It will email you as soon as logs are created.

# logs.watchr

#!/usr/bin/env ruby
# encoding: utf-8

require 'watchr'
require 'mail'

puts "Watching log files!"

# formats line of data from log
def tidy_line(l)
  # ...
end

# monitors logs in subdirectories
watch("logs/(.*?)/(.*?\.log)") do |f|
  # matches from regexp
  fpath, host, log = *f

  # parse log file
  str = ''
  File.foreach(fpath) do |line|
    str << tidy_line(line)
  end

  # email results
  Mail.deliver do
    to       "poutine@...com"
    from     "tacos@...com"
    subject  "LOG: #{ host } - #{ log }"
    body     str
  end
end