lognotify.rb

omp · 2010-10-05 05:45:14

http://github.com/omp/lognotify
Retrieve new lines from remote log files over SSH.

Information

The following information is straight from the header of the script:

# During the initial run, a cache file will be created and the entire log file
# will be retrieved. On any subsequent runs, only new lines will be retrieved
# and appended to the cached log.
#
# Although, besides for the addition of new lines, log files should never be
# altered, the remote log and the cached log are checked to be identical on
# every run, by comparing MD5 hashes. If either file has erroneously been
# altered, an error will be produced and the script will exit.

If you want to see the script in action, scroll down a bit. If you want to use it, please acquire it from the git repository I linked above, as I have not yet put out a versioned release. As for any feature requests or comments, this thread should be the proper place for them. I realise that this script would have a very limited audience, as most people would not have use for such a thing, but I figured it would be better to share it than to keep it to myself.

Why?

I recently started using irssi proxy. Upon connecting to the proxy via my local client, I wanted to know if I missed any important messages. Therefore, I decided to log noteworthy messages on the server and automate the process of acquiring new lines from the log each time I connected. I initially hacked together a quick shell script that did the job, but I decided to make it more complete and possibly release it in case anybody else had a use for such a script. In the process, I moved it over to a simple Ruby script, which does not require any third-party libraries.

Example

First, I create an example log file on the server:

omp@mycrosoft ~ $ echo -e "foo\nbar" > example.log
omp@mycrosoft ~ $ cat example.log 
foo
bar

Now, I create a simple configuration file locally:

omp@tard ~/dev/lognotify $ echo -e "hostname = mycrosoft.us\npath = ~/example.log" > ~/.config/lognotify/example.conf
omp@tard ~/dev/lognotify $ cat ~/.config/lognotify/example.conf 
hostname = mycrosoft.us
path = ~/example.log

Afterwards, I simply run the script with the identifier as an argument:

omp@tard ~/dev/lognotify $ ./lognotify.rb example
* Counting lines in cached log... 0
* Retrieving new lines via SSH... Done
* Number of new lines: 2
* Appending new lines to cached log... Done
foo
bar

If I run it once again, with the remote log file unchanged, it doesn't do much:

omp@tard ~/dev/lognotify $ ./lognotify.rb example
* Counting lines in cached log... 2
* Retrieving new lines via SSH... Done
* Number of new lines: 0

But if I modify the remote log file:

omp@mycrosoft ~ $ echo "baz" >> example.log 
omp@mycrosoft ~ $ cat example.log 
foo
bar
baz

It acquires new lines when I run it again:

omp@tard ~/dev/lognotify $ ./lognotify.rb example
* Counting lines in cached log... 2
* Retrieving new lines via SSH... Done
* Number of new lines: 1
* Appending new lines to cached log... Done
baz

Furthermore, the cached copy of the log is identical to that on the server:

omp@tard ~/dev/lognotify $ cat ~/.cache/lognotify/example.log 
foo
bar
baz

If I were to do something silly, such as modify the remote log file:

omp@mycrosoft ~ $ sed -i -e 's/foo/log files should never be altered/' example.log 
omp@mycrosoft ~ $ cat example.log 
log files should never be altered
bar
baz

An error would be produced on the next run, telling me to delete the cached log file:

omp@tard ~/dev/lognotify $ ./lognotify.rb example
* Counting lines in cached log... 3
* Retrieving new lines via SSH... Done
* Number of new lines: 0
./lognotify.rb:103:in `verify_hash': Hash check failed; delete cached log file. (RuntimeError)
    from ./lognotify.rb:102:in `open'
    from ./lognotify.rb:102:in `verify_hash'
    from ./lognotify.rb:141
    from ./lognotify.rb:123:in `each'
    from ./lognotify.rb:123

Tada! That should be the end of this demo. The comments at the top of the script better explain the configuration process.

awkwood · 2010-10-05 15:32:01

Cheers omp!

I have one suggestion: your count_lines method will be quite slow on large log files.
Rather than use readlines you can optimize the read operations like so:

line_count = 0
File.open(identifier.to_cache_path) do |f|
  while block = f.read(1024)
    line_count += block.count("\n")
  end
end

The speed boost makes it comparable to shelling out to wc.

Last edited by awkwood (2010-10-05 15:41:56)

omp · 2010-10-05 16:39:28

awkwood wrote:

Cheers omp!
I have one suggestion: your count_lines method will be quite slow on large log files.
Rather than use readlines you can optimize the read operations like so:
line_count = 0
File.open(identifier.to_cache_path) do |f|
  while block = f.read(1024)
    line_count += block.count("\n")
  end
end
The speed boost makes it comparable to shelling out to wc.

Thanks for the suggestion; I just committed it.

Arch Linux

#1 2010-10-05 05:45:14

lognotify.rb - Retrieve new lines from remote log files over SSH.

lognotify.rb

Information

Why?

Example

#2 2010-10-05 15:32:01

Re: lognotify.rb - Retrieve new lines from remote log files over SSH.

#3 2010-10-05 16:39:28

Re: lognotify.rb - Retrieve new lines from remote log files over SSH.

Board footer