Logfile issue

Benedict_White · 2007-09-19 21:32:17

I have a log file I need to pass to an analyser but it seems to be having a problem with 29 lines out of 5,715,031 lines.

The text that seems to be causing the problem is

DIGEST_MULTIPLE,NO_RDNS,PYZOR_CHECK,URIBL_AB_SURBL,URIBL_JP_SURBL,URIBL_SBL,URIBL_SC_SURBL

and I need to remove that tex, or if that is not possible the lines on which it occurs.

I have to say the log files are too big to open in text editors without crashing the server on which it is on.

With regret I am not much of a shell command sort of chap.

Can anyone help?

Benedict_White · 2007-09-20 01:11:40

Ok, I have figured out how to do that with sed. like this:

sed -i 's/DIGEST_MULTIPLE,NO_RDNS,PYZOR_CHECK,URIBL_AB_SURBL,URIBL_JP_SURBL,URIBL_SBL,URIBL_SC_SURBL/\ /g' /var/log/some.log

which as far as I can tell replaces that text with a space. However that has not fixd the problem.

Perhaps it is the file size.

So how can I split the file into chunks by say line number?

As in what tool could I use to say give me the first 3,000,000 lines of that file and put it in this file, then a way of saying put the rest in another file?

byte · 2007-09-20 06:13:37

You already were on your way...

sed.

sed -n '1,3000000p' huuuge_logfile > hugelog.1
sed -n '3000001,$p' huuuge_logfile > hugelog.2

Last edited by byte (2007-09-20 06:15:34)

Benedict_White · 2007-09-20 08:37:11

Many thanks byte, that looks to have done the trick.

I am obviously going to have to learn a lot more about sed.

Arch Linux

#1 2007-09-19 21:32:17

Logfile issue

#2 2007-09-20 01:11:40

Re: Logfile issue

#3 2007-09-20 06:13:37

Re: Logfile issue

#4 2007-09-20 08:37:11

Re: Logfile issue

Board footer