[SOLVED] big log sizes, what to do? - modified syslog-ng.conf

kaivalagi · 2010-02-02 11:19:28

Hi All,

I have noticied that my main logs are growing rapidly all the time, the below are the culprits, this is what they are like right now:

-rw-r-----  1 root log   1388693390 Feb  2 11:06 daemon.log
-rw-r-----  1 root log   1388712746 Feb  2 11:06 everything.log
-rw-r-----  1 root log   1388498797 Feb  2 11:06 messages.log

I have been clearing logs down using the following which I assume only removes archived files:

sudo rm /var/log/*.{1,2,3,4,5}

Firstly, is this size of logs normal?
Secondly, what would you recommend as the best maintenance plan going forwards, logrotate in cron? log deletion?

Any pointers to wiki articles? I have tried searching for maintenance help but can't seem to find anything useful...

Thanks in advance

Last edited by kaivalagi (2010-02-03 08:38:08)

brazzmonkey · 2010-02-02 11:25:00

a good starting point:
http://wiki.archlinux.org/index.php/NAT'ing_firewall_-_Adding_advanced_features#Control_those_logs_-_logrotate

Now, your logs are really huge. Maybe some programs should be configured to be less verbose...

Last edited by brazzmonkey (2010-02-02 11:26:39)

kaivalagi · 2010-02-02 12:03:14

brazzmonkey wrote:

a good starting point:
http://wiki.archlinux.org/index.php/NAT'ing_firewall_-_Adding_advanced_features#Control_those_logs_-_logrotate
Now, your logs are really huge. Maybe some programs should be configured to be less verbose...

Is that all there is in the wiki on this, a section within "NAT'ing firewall"? I find that a bit strange...thanks for pointing it out anyway.

I've scanned through the logrotate man pages and have now edited /etc/logrotate.conf to do a daily rotation to /var/log/archive with compression

I guess I need to start going through the conf files for all my daemons now to see what logging I can reduce...

Any other tips/tricks?

brazzmonkey · 2010-02-02 12:33:01

kaivalagi wrote:

Is that all there is in the wiki on this, a section within "NAT'ing firewall"? I find that a bit strange...thanks for pointing it out anyway.

Well, feel free to improve the situation. Contributions are welcome.
I realize Gentoo wiki's not repaired, hence the links are not working. Still, the web is full with logrotate examples.

I've scanned through the logrotate man pages and have now edited /etc/logrotate.conf to do a daily rotation to /var/log/archive with compression

Well done.

I guess I need to start going through the conf files for all my daemons now to see what logging I can reduce...
Any other tips/tricks?

Check your logs first, you may find some useless entries that would help you to identify which programs talk too much.

kaivalagi · 2010-02-02 13:05:45

brazzmonkey wrote:

kaivalagi wrote:
Is that all there is in the wiki on this, a section within "NAT'ing firewall"? I find that a bit strange...thanks for pointing it out anyway.
Well, feel free to improve the situation. Contributions are welcome.
I realize Gentoo wiki's not repaired, hence the links are not working. Still, the web is full with logrotate examples.

I've added a small mention about log maintenance to the "General Recommendations" section of the wiki for now (refered to by beginners guide appendix) as follows:

Log Maintenance
By default log files are rotated using "logrotate", this rotates existing log files to an alternatively named file (suffixed with a number), emptying the original log files. Users can edit it's behaviour via the /etc/logrotate.conf file. For example the rotate frequency can be altered and tarball compression and archive directories can be defined for any logs which are rotated, for more details take a look at the /etc/logrotate.conf configuration file and/or run "man logrotate"

Once I've made sense of my log file size and what is causing it I'll add further info around reduces log growth

Last edited by kaivalagi (2010-02-02 13:42:39)

pointone · 2010-02-02 16:35:29

I split your addition out into a separate logrotate article; please feel free to expand it!

SoleSoul · 2010-02-02 16:47:27

Hi.
The size of your logs is not normal.
There may be a repetitive error log which is being sent many times per second.
A quick look at the logs will tell you what the problem is.
Check and share your findings with us.

You can see my experience here: http://bbs.archlinux.org/viewtopic.php?id=79767

anti-destin · 2010-02-02 17:22:51

1. to clear logs, you can just run logrotate:

sudo logrotate -f /etc/logrotate.conf

2. you should check your logs and try to see why they are filling up so quickly. fix the underlying problem first.

3. configure your logrotate.conf. i have mine set to rotate logs daily. you can also try using the size option.

4. logrotate is run daily (see /etc/cron.daily). note: with the old version of dcron, you need to make sure that your computer is on when the cron job is scheduled to run. but with the latest version of dcron, there is no issue.

kaivalagi · 2010-02-02 18:39:35

Hi All

I now know what the big contributor to my logs is, it's an app out of the AUR called mpdcron, which is pumping music related info on each song change into all 3 logs, but more importantly the mpdcron process also logs failures to connect to mpd which is where the majority of the log size is coming from.

For now I have my logs rotating daily, with compression, and I can delay the excess from an archive sub-folder

I have raised the first ever issue with the developer, as I can't seem to find any options to turn down/off logging for mpdcron. Hopefully there is a simple answer to the excess logging...

Can anyone suggest anything I could do to stop the logging? The app is just started when I start a gnome session and hooks into my mpd daemon, I can easily affect the way it is called for example...

lucke · 2010-02-02 18:51:56

Fiddle with /etc/syslog-ng/syslog.conf - you should be able to direct mpdcron's output to a separate file, which shouldn't be a problem.

I myself use metalog instead of syslog-ng - among other things, it comes with log rotating support built-in.

kaivalagi · 2010-02-02 18:52:25

pointone wrote:

I split your addition out into a separate logrotate article; please feel free to expand it!

Thanks, I'll take a look and if I can add any value to it I will

kaivalagi · 2010-02-02 18:55:57

lucke wrote:

Fiddle with /etc/syslog-ng/syslog.conf - you should be able to direct mpdcron's output to a separate file, which shouldn't be a problem.
I myself use metalog instead of syslog-ng - among other things, it comes with log rotating support built-in.

I'm glad I hit this issue now, I would have never got to know how logging properly works...I'll take a look at the syslog options and at metalog which sounds interesting

Cheers

Edit: I tried setting up something in syslog-ng for logging redirection but it doesn't seem to have taken affect, I restarted both the syslog-ng and mpdcron daemons after conf changes, but no change...

I added these (in appropriate lines) to the syslog-ng.conf file:

#new log file for destination of mpdcron logging
destination d_mpdcron { file("/var/log/mpd/mpdcron.log"); };

# filter with match case of mpdcron
filter f_mpdcron { program("mpdcron"); };

# direct logging to the new file for logs matching the filter
log { source(src); filter(f_mpdcron); destination(d_mpdcron); };

Also "touch"ed the new mpdcron.log file so it's there to write to...

I'll reboot tonight at some point (got others things happening on the box right now), maybe that will kick start the settings into play...:/

An example of excessive logging in my messages.log (last.fm/libre.fm scrobbling activity):

Feb 2 19:12:38 towerpc1 mpdcron[11085]: [scrobbler] [libre.fm] sending 'now playing' notification
Feb 2 19:12:38 towerpc1 mpdcron[11085]: [scrobbler] [last.fm] sending 'now playing' notification
Feb 2 19:12:38 towerpc1 mpdcron[11085]: [scrobbler] [last.fm] OK
Feb 2 19:12:38 towerpc1 mpdcron[11085]: [scrobbler] [libre.fm] OK
Feb 2 19:12:39 towerpc1 mpdcron[11085]: [scrobbler] Song (Soldiers Of Jah Army - Strong For Them), id: 5, pos: 5 not played long enough, skipping
Feb 2 19:12:40 towerpc1 mpdcron[11085]: [scrobbler] Song (Toots And The Maytals - Gold And Silver), id: 52, pos: 52 not played long enough, skipping
Feb 2 19:12:40 towerpc1 mpdcron[11085]: [scrobbler] [libre.fm] sending 'now playing' notification
Feb 2 19:12:40 towerpc1 mpdcron[11085]: [scrobbler] [last.fm] sending 'now playing' notification
Feb 2 19:12:40 towerpc1 mpdcron[11085]: [scrobbler] [libre.fm] OK
Feb 2 19:12:40 towerpc1 mpdcron[11085]: [scrobbler] [last.fm] OK

Last edited by kaivalagi (2010-02-02 19:23:28)

sdolim · 2010-02-03 02:53:30

You may have to explicitly tell syslog-ng to stop logging mpdcron to daemon, messages, and everything (if you haven't already done so). Add 'and not program("mpdcron")' to the filter for those logs.

kaivalagi · 2010-02-03 08:23:43

sdolim wrote:

You may have to explicitly tell syslog-ng to stop logging mpdcron to daemon, messages, and everything (if you haven't already done so). Add 'and not program("mpdcron")' to the filter for those logs.

Thanks for that, I'll bear that in mind although I've got to the point now where it must really be the developers responsibility to put some logging options into the app so this can be controlled at source, and not have it write to the general system logs at all. I can see how messy this syslog-ng.conf file could get

Edit: just checked the logs and the mpdcron messages are now directed to the mpdcron.log file as I wanted, not sure whether I was tired and didn't realise or not...I'm going to try adding conditions for logging levels to the filter and see if that helps, I only want errors logged really anyway.

Thanks for all your help guys!

Edit2 (4/2/10): Ali Polatel, the author of mpdcron, has already updated the source to have logging level options in the config, defaulting to errors logged only

Last edited by kaivalagi (2010-02-04 21:45:21)

Arch Linux

#1 2010-02-02 11:19:28

[SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#2 2010-02-02 11:25:00

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#3 2010-02-02 12:03:14

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#4 2010-02-02 12:33:01

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#5 2010-02-02 13:05:45

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#6 2010-02-02 16:35:29

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#7 2010-02-02 16:47:27

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#8 2010-02-02 17:22:51

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#9 2010-02-02 18:39:35

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#10 2010-02-02 18:51:56

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#11 2010-02-02 18:52:25

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#12 2010-02-02 18:55:57

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#13 2010-02-03 02:53:30

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

#14 2010-02-03 08:23:43

Re: [SOLVED] big log sizes, what to do? - modified syslog-ng.conf

Board footer