Arch Linux

eDio

Member

From: Ukraine, Kyiv

Registered: 2008-12-02

Posts: 422

anylogmerge — utility to merge interleaved logs

Hi all.

Recently I had to investigate an issue in a clustered application, and thus had to analyze bunch of interleaved log files. But of course I had to merge them first for convenience of analysis.

Brief googling did not helped at once, so I decided to develop small utility that merges interleaved logs.
As I'm trying to learn Go language, I decided to develop this in Go.

Tool helped me with this task, so I decided to share it, and here it is — anylogmerge. Perhaps (I hope so), someone will find it useful.

Utility is capable of merging logs basing on some key. Key can be defined as a set of columns or with regular expression.

For example, consider some number of log files with the following structure

DEBUG 2012/12/31 23:59:59 Ding!
INFO 2013/01/01 00:00:00 Happy New Year!
Best wishes and goodluck
DEBUG 2013/01/01 00:00:01 The first second of 2013 year

Anylogmerge can merge such logs, so all log entries will appear chronologically

anylogmerge -s "^[A-Z]+\ ([0-9\/]+)\ ([0-9\:]+).*$" /var/log/ny/*.log

anylogmerge extracts date from each log entry with provided regular expression and compares extracted dates lexicographically (only this mode of comparison for now).

Full description is available on github and gitorious, where project is hosted.

https://github.com/edio/anylogmerge
https://gitorious.org/anylogmerge

AUR package:
https://aur.archlinux.org/packages/anylogmerge-git/

Last edited by eDio (2013-06-16 20:39:27)

---

github.com/edio