Processing logfiles in real time

rine · 2009-05-30 20:08:37

Hey,
Let's say I write this:

#!/usr/bin/env python
import sys

for line in sys.stdin:
     print line

and I have a file that is written, like

while true; do echo $(date) >> bla; sleep 1; done &

What I now want would be something like

tail -f bla | ./script.py

This is just to make it simple obviously. In the end what I want to do is store logs in a database. The script for that is almost finished, but I can't think of a good way to monitor the logfile all the time.

karol · 2009-05-30 21:23:45

http://linux.die.net/man/1/watch
I have no idea whether you need the python script or a way to continuously monitor sth. Please, specify what you need more precisely.

livibetter · 2009-05-30 21:40:31

rine wrote:

tail -f bla | ./script.py

The only improvement that I see is to move tail into Python script. Open the file (filename passes via command line argument), use file.readline(). If you want to do as `tail -F` does, just monitor the changes using `os.fstat()`, then reopen the file when stat.ST_INODE (and stat.ST_DEV) change.

I think you can also accept multiple logfile, with Python, it should not be hard to expand for that; and `-` for stdin.

Last edited by livibetter (2009-05-30 21:42:15)

rine · 2009-05-30 23:05:51

karol wrote:

http://linux.die.net/man/1/watch
I have no idea whether you need the python script or a way to continuously monitor sth. Please, specify what you need more precisely.

I just need something to continuosly monitor the log file. I thought I said that in my last sentence, sorry if I didn't make myself clear. My script reads line by line, does some regex work and stores the results in a database.

livibetter wrote:

rine wrote:
tail -f bla | ./script.py
The only improvement that I see is to move tail into Python script. Open the file (filename passes via command line argument), use file.readline(). If you want to do as `tail -F` does, just monitor the changes using `os.fstat()`, then reopen the file when stat.ST_INODE (and stat.ST_DEV) change.
I think you can also accept multiple logfile, with Python, it should not be hard to expand for that; and `-` for stdin.

Hm, I guess I could change the script to use a file instead of stdin. But wouldn't I then monitor st_mtime? I could open the file, loop forever and call os.readline() everytime st_mtime changes. But then I wouldn't know what happens when logrotate does it's work and my script still has the file opened. I don't want to reopen the file everytime for the file would then be read completely again (it could be about 50-100MB).

karol · 2009-05-30 23:10:37

My script reads line by line, does some regex work and stores the results in a database.

Than the 'watch' command may be to simple for your task.

livibetter · 2009-05-30 23:17:32

rine wrote:

Hm, I guess I could change the script to use a file instead of stdin. But wouldn't I then monitor st_mtime? I could open the file, loop forever and call os.readline() everytime st_mtime changes. But then I wouldn't know what happens when logrotate does it's work and my script still has the file opened. I don't want to reopen the file everytime for the file would then be read completely again (it could be about 50-100MB).

If you don't care the rotates, you don't need to monitor anything, just simply readline().

And, No, you don't need to reopen every time, only when INODE changes via os.fstat(filepath). When INODE changes, generally, it's a new empty file, because it's got rotated. You must reopen the new file, your old file object can't read anything from the new file after log file rotates if you don't reopen.

Just curios, what kind of thing does your script process? It also writes result to database... mind to share after you finish?

Last edited by livibetter (2009-05-30 23:23:29)

HashBox · 2009-05-31 02:09:17

You may want to look into using http://pyinotify.sourceforge.net/ rather than polling with os.fstat as it is more efficient, but that is a minor point

rine · 2009-05-31 02:17:37

livibetter wrote:

Just curios, what kind of thing does your script process? It also writes result to database... mind to share after you finish?

This is unfinished. Of the regexes, only pfull and precipient are used atm. What it does is store the startdate, enddate and messages in a database. Later, I will store more information in more fields. The logs are from qpsmtpd.

#!/usr/bin/env python

import sys # for stdin
import re # for regular expression match
import time # for date and time conversions
import MySQLdb
from datetime import datetime # for parsing and formatting date and time

startdates = {}
enddates = {}
msgs = {}
rcpts = {}

inputdtformat = "%a %b %d %H:%M:%S %Y" # Date and time format in the log file
outputdtformat = "%Y-%m-%d %H:%M:%S" # Date and time format in the output

mysql_opts = {
        'host': 'localhost',
        'user': 'qp',
        'pass': '####',
        'db':   'maillogs'}

pfull = re.compile(r'(?P<rdate>\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\s+\d+).*\[(?P<rid>\d*)\]:\s+(?P<rmessage>.*)$')
psender = re.compile(r'dispatching mail from:<[^@]*(?P<rsndr>\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b)', re.IGNORECASE)
precipient = re.compile(r'dispatching rcpt to:<[^@]*(?P<rrcpt>\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b)', re.IGNORECASE)
pspamass = re.compile(r'spamassassin//check_spam: (?P<isspam>Yes|No), hits=(?P<hits>\d+\.?\d+), required=(?P<req>\d+\.?\d+), tests=(?P<tests>.*)//spamassassin', re.IGNORECASE)
plogterse = re.compile(r'logging::logterse//`(?P<ip>.*?)\t(?P<hostname>.*?)\t(?P<helohost>.*?)\t(?P<envelopesender>.*?)\t(?P<recipient>.*?)\t(?P<plugin>.*?)\t(?P<return>.*?)\t(?P<deny>.*?$)', re.IGNORECASE)

# loop through lines
for line in sys.stdin:
    # match a single line of the log file
    m = pfull.match(line)
    if (m):
        date = m.group('rdate') # date and time of the entry as string
        id = m.group('rid') # process id
        msg = m.group('rmessage') # log message
        if id not in msgs: # if no entry yet for the process id
            # store the start date (converted to a datetime object)
            startdates[id] = datetime.fromtimestamp(time.mktime(time.strptime(date, inputdtformat)))
            msgs[id] = msg # store the log message
        else:
            msgs[id] = msgs[id] + "//" + msg # append the log message
        # store the end date
        enddates[id] = datetime.fromtimestamp(time.mktime(time.strptime(date, inputdtformat)))
# get recipients
for id in msgs:
    m = precipient.search(msgs[id])
    if (m):
        rcpts[id] = m.group('rrcpt')

ids = msgs.keys() # take all ids
ids.sort(lambda a,b: cmp(startdates[a], startdates[b])) # and sort them by start date

conn = MySQLdb.connect(mysql_opts['host'],
                        mysql_opts['user'],
                        mysql_opts['pass'],
                        mysql_opts['db'])
cursor = conn.cursor()
for id in ids:
        cursor.execute("INSERT INTO logs (pid, startdate, enddate, messages, recipient) VALUES (%s, %s, %s, %s, %s)", (id, startdates[id].strftime(outputdtformat), enddates[id].strftime(outputdtformat), msgs[id], rcpts[id] if id in rcpts else ''))

cursor.close()
conn.close()

If you want to test it, here is how you can create the table (create the empty database 'maillogs' first):

#!/usr/bin/env python
# Create or drop the log table
import sys
import MySQLdb

mcmd = sys.argv[1]
mysql_opts = { 
        'host': 'localhost',
        'user': 'qp',
        'pass': '###',
        'db':   'maillogs'}

conn = MySQLdb.connect(mysql_opts['host'],
                        mysql_opts['user'],
                        mysql_opts['pass'],
                        mysql_opts['db'])
cursor = conn.cursor()
if (mcmd == "create"):
    cursor.execute("""CREATE TABLE `maillogs`.`logs` (
                          `id` INT AUTO_INCREMENT,
                          `pid` VARCHAR(8) ,
                          `startdate` DATETIME ,
                          `enddate` DATETIME ,
                          `messages` MEDIUMTEXT ,
                          `recipient` VARCHAR(60),
                          PRIMARY KEY (`id`))""")
elif (mcmd == "drop"):
    cursor.execute('DROP TABLE `logs`;')
elif (mcmd == "-h" or mcmd == "--help"):
    print "Usage:", sys.argv[0], "create | drop"
else:
    print "Unknown command:", mcmd, "\nUse \"create\", \"drop\" or \"--help\""
cursor.close()
conn.close()

A sample logfile is here (I replaced most of the addresses):

Thu May 14 06:25:10 2009 server.server.com[19644]: Loaded Qpsmtpd::Plugin::logging::file=HASH(0x5e9830)
Thu May 14 06:25:10 2009 server.server.com[19644]: Loaded Qpsmtpd::Plugin::logging::logterse=HASH(0xfbbfd0)
Thu May 14 06:25:10 2009 server.server.com[19644]: Listening on port 25
Thu May 14 06:25:10 2009 server.server.com[19644]: Running as user qpsmtpd, group qpsmtpd
Thu May 14 06:25:10 2009 server.server.com[19647]: Initializing spool_dir
Thu May 14 06:25:10 2009 server.server.com[19647]: size_threshold set to 0
Thu May 14 06:25:19 2009 server.server.com[19804]: Accepted connection 0/15 from 124.43.42.218 / Unknown
Thu May 14 06:25:19 2009 server.server.com[19804]: Connection from Unknown [124.43.42.218]
Thu May 14 06:25:19 2009 server.server.com[19804]: logging::file
Thu May 14 06:25:19 2009 server.server.com[19804]: check_earlytalker
Thu May 14 06:25:20 2009 server.server.com[19804]: remote host said nothing spontaneous, proceeding
Thu May 14 06:25:20 2009 server.server.com[19804]: check_relay
Thu May 14 06:25:20 2009 server.server.com[19804]: dnsbl
Thu May 14 06:25:20 2009 server.server.com[19804]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:25:20 2009 server.server.com[19804]: dispatching ehlo digitalbunny.com
Thu May 14 06:25:20 2009 server.server.com[19804]: tls
Thu May 14 06:25:20 2009 server.server.com[19804]: check_spamhelo
Thu May 14 06:25:20 2009 server.server.com[19804]: 250-mail.test.com Hi Unknown [124.43.42.218]
Thu May 14 06:25:20 2009 server.server.com[19804]: 250-PIPELINING
Thu May 14 06:25:20 2009 server.server.com[19804]: 250-8BITMIME
Thu May 14 06:25:20 2009 server.server.com[19804]: 250-STARTTLS
Thu May 14 06:25:20 2009 server.server.com[19804]: 250 AUTH LOGIN CRAM-MD5
Thu May 14 06:25:21 2009 server.server.com[19805]: Accepted connection 1/15 from 93.94.233.84 / Unknown
Thu May 14 06:25:21 2009 server.server.com[19805]: Connection from Unknown [93.94.233.84]
Thu May 14 06:25:21 2009 server.server.com[19805]: logging::file
Thu May 14 06:25:21 2009 server.server.com[19805]: check_earlytalker
Thu May 14 06:25:21 2009 server.server.com[19804]: dispatching mail from:<test@test.com>
Thu May 14 06:25:21 2009 server.server.com[19804]: full from_parameter: from:<test@test.com>
Thu May 14 06:25:21 2009 server.server.com[19804]: from email address : [<test@test.com>]
Thu May 14 06:25:21 2009 server.server.com[19804]: tls
Thu May 14 06:25:21 2009 server.server.com[19804]: rhsbl
Thu May 14 06:25:21 2009 server.server.com[19804]: check_badmailfrom
Thu May 14 06:25:21 2009 server.server.com[19804]: getting mail from <test@test.com>
Thu May 14 06:25:21 2009 server.server.com[19804]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:25:21 2009 server.server.com[19804]: dispatching rcpt to:<test@test.com>
Thu May 14 06:25:21 2009 server.server.com[19804]: to email address : [<test@test.com>]
Thu May 14 06:25:21 2009 server.server.com[19804]: tls
Thu May 14 06:25:21 2009 server.server.com[19804]: rhsbl
Thu May 14 06:25:21 2009 server.server.com[19804]: dnsbl
Thu May 14 06:25:21 2009 server.server.com[19804]: check_badmailfrom
Thu May 14 06:25:21 2009 server.server.com[19804]: check_badrcptto
Thu May 14 06:25:21 2009 server.server.com[19804]: rcpt_ok
Thu May 14 06:25:21 2009 server.server.com[19804]: 250 <test@test.com>, recipient ok
Thu May 14 06:25:22 2009 server.server.com[19804]: dispatching data
Thu May 14 06:25:22 2009 server.server.com[19804]: tls
Thu May 14 06:25:22 2009 server.server.com[19804]: 354 go ahead
Thu May 14 06:25:22 2009 server.server.com[19805]: remote host said nothing spontaneous, proceeding
Thu May 14 06:25:22 2009 server.server.com[19805]: check_relay
Thu May 14 06:25:22 2009 server.server.com[19805]: dnsbl
Thu May 14 06:25:22 2009 server.server.com[19805]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:25:22 2009 server.server.com[19805]: dispatching HELO 93.94.233.84
Thu May 14 06:25:22 2009 server.server.com[19805]: tls
Thu May 14 06:25:22 2009 server.server.com[19805]: check_spamhelo
Thu May 14 06:25:22 2009 server.server.com[19805]: 250 mail.test.com Hi Unknown [93.94.233.84]; I am so happy to meet you.
Thu May 14 06:25:22 2009 server.server.com[19805]: dispatching MAIL FROM: <test@test.com>
Thu May 14 06:25:22 2009 server.server.com[19805]: full from_parameter: FROM: <test@test.com>
Thu May 14 06:25:22 2009 server.server.com[19805]: from email address : [<test@test.com>]
Thu May 14 06:25:22 2009 server.server.com[19805]: tls
Thu May 14 06:25:22 2009 server.server.com[19805]: rhsbl
Thu May 14 06:25:22 2009 server.server.com[19805]: check_badmailfrom
Thu May 14 06:25:22 2009 server.server.com[19805]: getting mail from <test@test.com>
Thu May 14 06:25:22 2009 server.server.com[19805]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:25:22 2009 server.server.com[19805]: dispatching RCPT TO: <test@test.com>
Thu May 14 06:25:22 2009 server.server.com[19805]: to email address : [<test@test.com>]
Thu May 14 06:25:22 2009 server.server.com[19805]: tls
Thu May 14 06:25:22 2009 server.server.com[19805]: rhsbl
Thu May 14 06:25:22 2009 server.server.com[19805]: dnsbl
Thu May 14 06:25:22 2009 server.server.com[19805]: check_badmailfrom
Thu May 14 06:25:22 2009 server.server.com[19805]: check_badrcptto
Thu May 14 06:25:22 2009 server.server.com[19805]: rcpt_ok
Thu May 14 06:25:22 2009 server.server.com[19805]: 250 <test@test.com>, recipient ok
Thu May 14 06:25:22 2009 server.server.com[19805]: dispatching DATA
Thu May 14 06:25:22 2009 server.server.com[19805]: tls
Thu May 14 06:25:22 2009 server.server.com[19805]: 354 go ahead
Thu May 14 06:25:22 2009 server.server.com[19804]: check_loop
Thu May 14 06:25:22 2009 server.server.com[19804]: spamassassin
Thu May 14 06:25:22 2009 server.server.com[19805]: check_loop
Thu May 14 06:25:22 2009 server.server.com[19805]: spamassassin
Thu May 14 06:25:30 2009 server.server.com[19804]: check_spam: Yes, hits=10.3, required=3.5, tests=BAYES_99,PYZOR_CHECK,RCVD_IN_JMF_BL,RDNS_NONE
Thu May 14 06:25:30 2009 server.server.com[19804]: spamassassin
Thu May 14 06:25:30 2009 server.server.com[19804]: logging::logterse
Thu May 14 06:25:30 2009 server.server.com[19804]: `124.43.42.218    Unknown    digitalbunny.com    <test@test.com>    
Thu May 14 06:25:30 2009 server.server.com[19804]: 552 spam score exceeded threshold (#5.6.1)
Thu May 14 06:25:32 2009 server.server.com[19647]: cleaning up after 19804
Thu May 14 06:25:34 2009 server.server.com[19805]: check_spam: Yes, hits=23.1, required=3.5, tests=AWL,BAYES_99,HTML_MESSAGE,MIME_HTML_ONLY,MISSING_DATE,MISSING_MID,NIX_SPAM,PYZOR_CHECK,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_JMF_BL,RCVD_NUMERIC_HELO,RDNS_NONE,URIBL_BLACK,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_WS_SURBL
Thu May 14 06:25:34 2009 server.server.com[19805]: spamassassin
Thu May 14 06:25:34 2009 server.server.com[19805]: logging::logterse
Thu May 14 06:25:34 2009 server.server.com[19805]: `93.94.233.84    Unknown    93.94.233.84    <test@test.com>    
Thu May 14 06:25:34 2009 server.server.com[19805]: 552 spam score exceeded threshold (#5.6.1)
Thu May 14 06:25:35 2009 server.server.com[19647]: cleaning up after 19805
Thu May 14 06:25:41 2009 server.server.com[19809]: Accepted connection 0/15 from 189.73.229.66 / 189-73-229-66.dosce700.dsl.brasiltelecom.net.br
Thu May 14 06:25:41 2009 server.server.com[19809]: Connection from 189-73-229-66.dosce700.dsl.brasiltelecom.net.br [189.73.229.66]
Thu May 14 06:25:41 2009 server.server.com[19809]: logging::file
Thu May 14 06:25:41 2009 server.server.com[19809]: check_earlytalker
Thu May 14 06:25:42 2009 server.server.com[19809]: remote host said nothing spontaneous, proceeding
Thu May 14 06:25:42 2009 server.server.com[19809]: check_relay
Thu May 14 06:25:42 2009 server.server.com[19809]: dnsbl
Thu May 14 06:25:42 2009 server.server.com[19809]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:25:42 2009 server.server.com[19809]: dispatching HELO 189.73.229.66
Thu May 14 06:25:42 2009 server.server.com[19809]: tls
Thu May 14 06:25:42 2009 server.server.com[19809]: check_spamhelo
Thu May 14 06:25:42 2009 server.server.com[19809]: 250 mail.test.com Hi 189-73-229-66.dosce700.dsl.brasiltelecom.net.br [189.73.229.66]; I am so happy to meet you.
Thu May 14 06:25:42 2009 server.server.com[19809]: dispatching MAIL FROM: <test@test.com>
Thu May 14 06:25:42 2009 server.server.com[19809]: full from_parameter: FROM: <test@test.com>
Thu May 14 06:25:42 2009 server.server.com[19809]: from email address : [<test@test.com>]
Thu May 14 06:25:42 2009 server.server.com[19809]: tls
Thu May 14 06:25:42 2009 server.server.com[19809]: rhsbl
Thu May 14 06:25:42 2009 server.server.com[19809]: check_badmailfrom
Thu May 14 06:25:42 2009 server.server.com[19809]: getting mail from <test@test.com>
Thu May 14 06:25:42 2009 server.server.com[19809]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:25:42 2009 server.server.com[19809]: dispatching RCPT TO: <test@test.com>
Thu May 14 06:25:42 2009 server.server.com[19809]: to email address : [<test@test.com>]
Thu May 14 06:25:42 2009 server.server.com[19809]: tls
Thu May 14 06:25:42 2009 server.server.com[19809]: rhsbl
Thu May 14 06:25:42 2009 server.server.com[19809]: dnsbl
Thu May 14 06:25:42 2009 server.server.com[19809]: check_badmailfrom
Thu May 14 06:25:42 2009 server.server.com[19809]: check_badrcptto
Thu May 14 06:25:42 2009 server.server.com[19809]: rcpt_ok
Thu May 14 06:25:42 2009 server.server.com[19809]: 250 <test@test.com>, recipient ok
Thu May 14 06:25:43 2009 server.server.com[19809]: dispatching DATA
Thu May 14 06:25:43 2009 server.server.com[19809]: tls
Thu May 14 06:25:43 2009 server.server.com[19809]: 354 go ahead
Thu May 14 06:25:43 2009 server.server.com[19809]: check_loop
Thu May 14 06:25:43 2009 server.server.com[19809]: spamassassin
Thu May 14 06:25:55 2009 server.server.com[19809]: check_spam: Yes, hits=21.7, required=3.5, tests=BAYES_99,HTML_MESSAGE,MIME_HTML_ONLY,MISSING_DATE,MISSING_MID,NIX_SPAM,PYZOR_CHECK,RCVD_IN_JMF_BL,RCVD_NUMERIC_HELO,RDNS_DYNAMIC,TVD_RCVD_IP,URIBL_BLACK,URIBL_JP_SURBL,URIBL_WS_SURBL
Thu May 14 06:25:55 2009 server.server.com[19809]: spamassassin
Thu May 14 06:25:55 2009 server.server.com[19809]: logging::logterse
Thu May 14 06:25:55 2009 server.server.com[19809]: `189.73.229.66    189-73-229-66.dosce700.dsl.brasiltelecom.net.br    189.73.229.66    <test@test.com>    
Thu May 14 06:25:55 2009 server.server.com[19809]: 552 spam score exceeded threshold (#5.6.1)
Thu May 14 06:25:56 2009 server.server.com[19647]: cleaning up after 19809
Thu May 14 06:26:02 2009 server.server.com[19811]: Accepted connection 0/15 from 59.92.125.91 / Unknown
Thu May 14 06:26:02 2009 server.server.com[19811]: Connection from Unknown [59.92.125.91]
Thu May 14 06:26:02 2009 server.server.com[19811]: logging::file
Thu May 14 06:26:02 2009 server.server.com[19811]: check_earlytalker
Thu May 14 06:26:03 2009 server.server.com[19811]: remote host said nothing spontaneous, proceeding
Thu May 14 06:26:03 2009 server.server.com[19811]: check_relay
Thu May 14 06:26:03 2009 server.server.com[19811]: dnsbl
Thu May 14 06:26:03 2009 server.server.com[19811]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:26:03 2009 server.server.com[19811]: dispatching EHLO [59.96.23.222]
Thu May 14 06:26:03 2009 server.server.com[19811]: tls
Thu May 14 06:26:04 2009 server.server.com[19811]: check_spamhelo
Thu May 14 06:26:04 2009 server.server.com[19811]: 250-mail.test.com Hi Unknown [59.92.125.91]
Thu May 14 06:26:04 2009 server.server.com[19811]: 250-PIPELINING
Thu May 14 06:26:04 2009 server.server.com[19811]: 250-8BITMIME
Thu May 14 06:26:04 2009 server.server.com[19811]: 250-STARTTLS
Thu May 14 06:26:04 2009 server.server.com[19811]: 250 AUTH LOGIN CRAM-MD5
Thu May 14 06:26:04 2009 server.server.com[19811]: dispatching MAIL FROM:<test@test.com>
Thu May 14 06:26:04 2009 server.server.com[19811]: full from_parameter: FROM:<test@test.com>
Thu May 14 06:26:04 2009 server.server.com[19811]: from email address : [<test@test.com>]
Thu May 14 06:26:04 2009 server.server.com[19811]: tls
Thu May 14 06:26:04 2009 server.server.com[19811]: rhsbl
Thu May 14 06:26:04 2009 server.server.com[19811]: check_badmailfrom
Thu May 14 06:26:04 2009 server.server.com[19811]: getting mail from <test@test.com>
Thu May 14 06:26:04 2009 server.server.com[19811]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:26:05 2009 server.server.com[19811]: dispatching RCPT TO:<test@test.com>
Thu May 14 06:26:05 2009 server.server.com[19811]: to email address : [<test@test.com>]
Thu May 14 06:26:05 2009 server.server.com[19811]: tls
Thu May 14 06:26:05 2009 server.server.com[19811]: rhsbl
Thu May 14 06:26:05 2009 server.server.com[19811]: dnsbl
Thu May 14 06:26:05 2009 server.server.com[19811]: check_badmailfrom
Thu May 14 06:26:05 2009 server.server.com[19811]: check_badrcptto
Thu May 14 06:26:05 2009 server.server.com[19811]: rcpt_ok
Thu May 14 06:26:05 2009 server.server.com[19811]: 250 <test@test.com>, recipient ok
Thu May 14 06:26:05 2009 server.server.com[19811]: dispatching DATA
Thu May 14 06:26:05 2009 server.server.com[19811]: tls
Thu May 14 06:26:05 2009 server.server.com[19811]: 354 go ahead
Thu May 14 06:26:06 2009 server.server.com[19811]: check_loop
Thu May 14 06:26:06 2009 server.server.com[19811]: spamassassin
Thu May 14 06:26:10 2009 server.server.com[19812]: Accepted connection 1/15 from 196.12.245.144 / wana-144-245-12-196.wanamaroc.com
Thu May 14 06:26:10 2009 server.server.com[19812]: Connection from wana-144-245-12-196.wanamaroc.com [196.12.245.144]
Thu May 14 06:26:10 2009 server.server.com[19812]: logging::file
Thu May 14 06:26:10 2009 server.server.com[19812]: check_earlytalker
Thu May 14 06:26:10 2009 server.server.com[19813]: Accepted connection 2/15 from 121.157.121.27 / Unknown
Thu May 14 06:26:10 2009 server.server.com[19813]: Connection from Unknown [121.157.121.27]
Thu May 14 06:26:10 2009 server.server.com[19813]: logging::file
Thu May 14 06:26:10 2009 server.server.com[19813]: check_earlytalker
Thu May 14 06:26:11 2009 server.server.com[19812]: remote host said nothing spontaneous, proceeding
Thu May 14 06:26:11 2009 server.server.com[19812]: check_relay
Thu May 14 06:26:11 2009 server.server.com[19812]: dnsbl
Thu May 14 06:26:11 2009 server.server.com[19812]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:26:11 2009 server.server.com[19812]: dispatching ehlo wana-144-245-12-196.wanamaroc.com
Thu May 14 06:26:11 2009 server.server.com[19812]: tls
Thu May 14 06:26:11 2009 server.server.com[19812]: check_spamhelo
Thu May 14 06:26:11 2009 server.server.com[19812]: 250-mail.test.com Hi wana-144-245-12-196.wanamaroc.com [196.12.245.144]
Thu May 14 06:26:11 2009 server.server.com[19812]: 250-PIPELINING
Thu May 14 06:26:11 2009 server.server.com[19812]: 250-8BITMIME
Thu May 14 06:26:11 2009 server.server.com[19812]: 250-STARTTLS
Thu May 14 06:26:11 2009 server.server.com[19812]: 250 AUTH LOGIN CRAM-MD5
Thu May 14 06:26:11 2009 server.server.com[19813]: remote host said nothing spontaneous, proceeding
Thu May 14 06:26:11 2009 server.server.com[19813]: check_relay
Thu May 14 06:26:11 2009 server.server.com[19813]: dnsbl
Thu May 14 06:26:11 2009 server.server.com[19813]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:26:11 2009 server.server.com[19812]: dispatching mail from:<test@test.com>
Thu May 14 06:26:11 2009 server.server.com[19812]: full from_parameter: from:<test@test.com>
Thu May 14 06:26:11 2009 server.server.com[19812]: from email address : [<test@test.com>]
Thu May 14 06:26:11 2009 server.server.com[19812]: tls
Thu May 14 06:26:11 2009 server.server.com[19812]: rhsbl
Thu May 14 06:26:11 2009 server.server.com[19812]: check_badmailfrom
Thu May 14 06:26:11 2009 server.server.com[19812]: getting mail from <test@test.com>
Thu May 14 06:26:11 2009 server.server.com[19812]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:26:12 2009 server.server.com[19813]: dispatching EHLO RQWIWLNHNW
Thu May 14 06:26:12 2009 server.server.com[19813]: tls
Thu May 14 06:26:12 2009 server.server.com[19813]: check_spamhelo
Thu May 14 06:26:12 2009 server.server.com[19813]: 250-mail.test.com Hi Unknown [121.157.121.27]
Thu May 14 06:26:12 2009 server.server.com[19813]: 250-PIPELINING
Thu May 14 06:26:12 2009 server.server.com[19813]: 250-8BITMIME
Thu May 14 06:26:12 2009 server.server.com[19813]: 250-STARTTLS
Thu May 14 06:26:12 2009 server.server.com[19813]: 250 AUTH LOGIN CRAM-MD5
Thu May 14 06:26:12 2009 server.server.com[19812]: dispatching rcpt to:<test@test.com>
Thu May 14 06:26:12 2009 server.server.com[19812]: to email address : [<test@test.com>]
Thu May 14 06:26:12 2009 server.server.com[19812]: tls
Thu May 14 06:26:12 2009 server.server.com[19812]: rhsbl
Thu May 14 06:26:12 2009 server.server.com[19812]: dnsbl
Thu May 14 06:26:12 2009 server.server.com[19812]: check_badmailfrom
Thu May 14 06:26:12 2009 server.server.com[19812]: check_badrcptto
Thu May 14 06:26:12 2009 server.server.com[19812]: rcpt_ok
Thu May 14 06:26:12 2009 server.server.com[19812]: 250 <test@test.com>, recipient ok
Thu May 14 06:26:12 2009 server.server.com[19812]: dispatching data
Thu May 14 06:26:12 2009 server.server.com[19812]: tls
Thu May 14 06:26:12 2009 server.server.com[19812]: 354 go ahead
Thu May 14 06:26:12 2009 server.server.com[19812]: check_loop
Thu May 14 06:26:12 2009 server.server.com[19812]: spamassassin
Thu May 14 06:26:13 2009 server.server.com[19813]: dispatching MAIL FROM: <test@test.com>
Thu May 14 06:26:13 2009 server.server.com[19813]: full from_parameter: FROM: <test@test.com>
Thu May 14 06:26:13 2009 server.server.com[19813]: from email address : [<test@test.com>]
Thu May 14 06:26:13 2009 server.server.com[19813]: tls
Thu May 14 06:26:13 2009 server.server.com[19813]: rhsbl
Thu May 14 06:26:13 2009 server.server.com[19813]: check_badmailfrom
Thu May 14 06:26:13 2009 server.server.com[19813]: getting mail from <test@test.com>
Thu May 14 06:26:13 2009 server.server.com[19813]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:26:13 2009 server.server.com[19813]: dispatching RCPT TO: <test@test.com>
Thu May 14 06:26:13 2009 server.server.com[19813]: to email address : [<test@test.com>]
Thu May 14 06:26:13 2009 server.server.com[19813]: tls
Thu May 14 06:26:13 2009 server.server.com[19813]: rhsbl
Thu May 14 06:26:13 2009 server.server.com[19813]: dnsbl
Thu May 14 06:26:13 2009 server.server.com[19813]: check_badmailfrom
Thu May 14 06:26:13 2009 server.server.com[19813]: check_badrcptto
Thu May 14 06:26:13 2009 server.server.com[19813]: rcpt_ok
Thu May 14 06:26:13 2009 server.server.com[19813]: 250 <test@test.com>, recipient ok
Thu May 14 06:26:13 2009 server.server.com[19813]: dispatching DATA
Thu May 14 06:26:13 2009 server.server.com[19813]: tls
Thu May 14 06:26:13 2009 server.server.com[19813]: 354 go ahead
Thu May 14 06:26:14 2009 server.server.com[19813]: check_loop
Thu May 14 06:26:14 2009 server.server.com[19813]: spamassassin
Thu May 14 06:26:20 2009 server.server.com[19816]: Accepted connection 3/15 from 159.153.156.60 / Unknown
Thu May 14 06:26:20 2009 server.server.com[19816]: Connection from Unknown [159.153.156.60]
Thu May 14 06:26:20 2009 server.server.com[19816]: logging::file
Thu May 14 06:26:20 2009 server.server.com[19816]: check_earlytalker
Thu May 14 06:26:20 2009 server.server.com[19812]: check_spam: Yes, hits=14.7, required=3.5, tests=BAYES_99,FH_HELO_EQ_D_D_D_D,FM_SEX_HELODDDD,PYZOR_CHECK,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_JMF_BL,RCVD_IN_SORBS_WEB,RDNS_DYNAMIC
Thu May 14 06:26:20 2009 server.server.com[19812]: spamassassin
Thu May 14 06:26:20 2009 server.server.com[19812]: logging::logterse
Thu May 14 06:26:20 2009 server.server.com[19812]: `196.12.245.144    wana-144-245-12-196.wanamaroc.com    wana-144-245-12-196.wanamaroc.com    <test@test.com>    
Thu May 14 06:26:20 2009 server.server.com[19812]: 552 spam score exceeded threshold (#5.6.1)
Thu May 14 06:26:21 2009 server.server.com[19816]: remote host said nothing spontaneous, proceeding
Thu May 14 06:26:21 2009 server.server.com[19816]: check_relay
Thu May 14 06:26:21 2009 server.server.com[19647]: cleaning up after 19812
Thu May 14 06:26:21 2009 server.server.com[19816]: dnsbl
Thu May 14 06:26:21 2009 server.server.com[19816]: 220 mail.test.com ESMTP qpsmtpd 0.32 ready; send us your mail, but not your spam.
Thu May 14 06:26:21 2009 server.server.com[19816]: dispatching EHLO chello089076039054.chello.pl
Thu May 14 06:26:21 2009 server.server.com[19816]: tls
Thu May 14 06:26:21 2009 server.server.com[19816]: check_spamhelo
Thu May 14 06:26:21 2009 server.server.com[19816]: 250-mail.test.com Hi Unknown [159.153.156.60]
Thu May 14 06:26:21 2009 server.server.com[19816]: 250-PIPELINING
Thu May 14 06:26:21 2009 server.server.com[19816]: 250-8BITMIME
Thu May 14 06:26:21 2009 server.server.com[19816]: 250-STARTTLS
Thu May 14 06:26:21 2009 server.server.com[19816]: 250 AUTH LOGIN CRAM-MD5
Thu May 14 06:26:21 2009 server.server.com[19816]: dispatching MAIL FROM:<test@test.com>
Thu May 14 06:26:21 2009 server.server.com[19816]: full from_parameter: FROM:<test@test.com>
Thu May 14 06:26:21 2009 server.server.com[19816]: from email address : [<test@test.com>]
Thu May 14 06:26:21 2009 server.server.com[19816]: tls
Thu May 14 06:26:21 2009 server.server.com[19816]: rhsbl
Thu May 14 06:26:21 2009 server.server.com[19816]: check_badmailfrom
Thu May 14 06:26:21 2009 server.server.com[19816]: getting mail from <test@test.com>
Thu May 14 06:26:21 2009 server.server.com[19816]: 250 <test@test.com>, sender OK - how exciting to get mail from you!
Thu May 14 06:26:21 2009 server.server.com[19816]: dispatching RCPT TO:<test@test.com>
Thu May 14 06:26:21 2009 server.server.com[19816]: to email address : [<test@test.com>]
Thu May 14 06:26:21 2009 server.server.com[19816]: tls
Thu May 14 06:26:21 2009 server.server.com[19816]: rhsbl
Thu May 14 06:26:21 2009 server.server.com[19816]: dnsbl
Thu May 14 06:26:21 2009 server.server.com[19816]: check_badmailfrom
Thu May 14 06:26:21 2009 server.server.com[19816]: check_badrcptto
Thu May 14 06:26:21 2009 server.server.com[19816]: rcpt_ok
Thu May 14 06:26:21 2009 server.server.com[19816]: 250 <test@test.com>, recipient ok
Thu May 14 06:26:21 2009 server.server.com[19816]: dispatching DATA
Thu May 14 06:26:21 2009 server.server.com[19816]: tls
Thu May 14 06:26:21 2009 server.server.com[19816]: 354 go ahead
Thu May 14 06:26:21 2009 server.server.com[19816]: check_loop
Thu May 14 06:26:21 2009 server.server.com[19816]: spamassassin

Last edited by rine (2009-05-31 12:36:07)

Arch Linux

#1 2009-05-30 20:08:37

Processing logfiles in real time

#2 2009-05-30 21:23:45

Re: Processing logfiles in real time

#3 2009-05-30 21:40:31

Re: Processing logfiles in real time

#4 2009-05-30 23:05:51

Re: Processing logfiles in real time

#5 2009-05-30 23:10:37

Re: Processing logfiles in real time

#6 2009-05-30 23:17:32

Re: Processing logfiles in real time

#7 2009-05-31 02:09:17

Re: Processing logfiles in real time

#8 2009-05-31 02:17:37

Re: Processing logfiles in real time

Board footer