[SOLVED] syslog-ng using 100% CPU

lb.laboon · 2014-12-08 14:40:35

A little after midnight today I noticed that syslog-ng suddenly started using 100% of my system's CPU. I let it go for a little bit thinking it would eventually stop, but after 10 minutes it was still going. I stopped the syslog-ng service and everything was fine, but when I started it again (and reloaded the daemon) it went right back to using 100% CPU. I tried stopping all the services I could think of that might be logging to syslog (including sshd), but that didn't do anything. I then tried rebooting, which also did not work. At this point I thought it might be a bug with syslog-ng, so I downgraded to the next-most-recent version in my pacman cache (3.5.6-1) and restarted the service, and that fixed the issue.

I saw someone posted a topic a while ago who was having the same issue, and it turned out to be an issue with their configuration. In my case, I'm using the standard configuration that comes with the package and I made sure there weren't any .pacnew files that needed merging.

Is anybody else seeing this issue with any of their systems?

Last edited by lb.laboon (2014-12-12 16:20:39)

Spider.007 · 2014-12-08 15:33:57

You could start the process and then use strace to get an idea of what it's doing. It might help you find the cause; or enough data to report a bug upstream

karol · 2014-12-08 15:39:14

Are you sure it's not https://bbs.archlinux.org/viewtopic.php?id=190701 ?

lb.laboon · 2014-12-08 16:21:45

karol wrote:

Are you sure it's not https://bbs.archlinux.org/viewtopic.php?id=190701 ?

Forgot to mention, I already checked that. My /etc/systemd/journald.conf contains #ForwardToSyslog=no, so I don't think that's the issue.

Spider.007 wrote:

You could start the process and then use strace to get an idea of what it's doing. It might help you find the cause; or enough data to report a bug upstream

I just reinstalled the latest syslog-ng and ran an strace on it. It's spewing syscalls like nobody's business. Looks like it's mostly the same ones being repeated. Here's a sample:

fcntl(12, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
fcntl(12, F_GETFL)                      = 0x800 (flags O_RDONLY|O_NONBLOCK)
setsockopt(12, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket)
write(80, "\1\0\0\0\0\0\0\0", 8)        = 8
epoll_wait(4, {{EPOLLIN, {u32=34926024, u64=34926024}}}, 6, 9754) = 1
read(5, "\1\0\0\0\0\0\0\0", 8)          = 8
epoll_wait(4, {{EPOLLIN, {u32=34926024, u64=34926024}}}, 6, 9753) = 1
read(5, "\1\0\0\0\0\0\0\0", 8)          = 8

I'm not sure if any of that's actually useful or not. There are tons of those ENOTSOCK errors, so maybe that's it? I ran an strace on the older (working) syslog-ng and it has practically nothing going on by comparison.

Last edited by lb.laboon (2014-12-08 16:27:00)

nostalgix · 2014-12-08 19:03:36

I came here because I was searching for exactly the same issue. My laptop is unusable right now, because of the syslog-ng's high cpu usage. And I am using it with default settings, always have.

karol · 2014-12-08 21:44:59

Try setting (uncommenting)

ForwardToSyslog=no

to explicitly disable it.

ihrwein · 2014-12-09 10:47:00

An Arch user reported a bug which looks like yours. The patch which solves his problem was integrated on the 3.6/master branch. Could you try to build syslog-ng on your PC from the upstream repo's 3.6/master branch? (https://github.com/balabit/syslog-ng.git)

If it doesn't work for you, please specify the versions of syslog-ng you tried and your configuration. Do you use systemd?

If you need help in the building, I can elaborate the process.

Last edited by ihrwein (2014-12-09 10:48:35)

lb.laboon · 2014-12-09 19:10:53

karol wrote:

Try setting (uncommenting)
ForwardToSyslog=no
to explicitly disable it.

That didn't change anything. I think "no" is the default.

ihrwein wrote:

An Arch user reported a bug which looks like yours. The patch which solves his problem was integrated on the 3.6/master branch. Could you try to build syslog-ng on your PC from the upstream repo's 3.6/master branch? (https://github.com/balabit/syslog-ng.git)
If it doesn't work for you, please specify the versions of syslog-ng you tried and your configuration. Do you use systemd?
If you need help in the building, I can elaborate the process.

I checked out and compiled the latest in the 3.6/master branch (commit 353852dd26) and that also didn't fix the issue. I first noticed the issue with version 3.6.1-1, but it only started happening yesterday after midnight. I've had it installed since it was released about a week ago. The only thing I've done so far that alleviates the issue is reverting back to the 3.5.6-1 release (I haven't checked if older versions work). I haven't made any modifications to my configuration, so all the defaults are being used. And yes, I do use systemd.

EDIT: Here's my syslog-ng.conf, since it looks like Arch's default config is different than what's shipped with the source.

@version: 3.5
@include "scl.conf"
#
# /etc/syslog-ng/syslog-ng.conf
#

options {
  stats_freq (0);
  flush_lines (0);
  time_reopen (10);
  log_fifo_size (10000);
  chain_hostnames (off);
  use_dns (no);
  use_fqdn (no);
  create_dirs (no);
  keep_hostname (yes);
  perm(0640);
  group("log");
};

source src {
  system();
  internal();
};

destination d_authlog { file("/var/log/auth.log"); };
destination d_syslog { file("/var/log/syslog.log"); };
destination d_cron { file("/var/log/crond.log"); };
destination d_daemon { file("/var/log/daemon.log"); };
destination d_kernel { file("/var/log/kernel.log"); };
destination d_lpr { file("/var/log/lpr.log"); };
destination d_user { file("/var/log/user.log"); };
destination d_uucp { file("/var/log/uucp.log"); };
destination d_mail { file("/var/log/mail.log"); };
destination d_news { file("/var/log/news.log"); };
destination d_ppp { file("/var/log/ppp.log"); };
destination d_debug { file("/var/log/debug.log"); };
destination d_messages { file("/var/log/messages.log"); };
destination d_errors { file("/var/log/errors.log"); };
destination d_everything { file("/var/log/everything.log"); };
destination d_iptables { file("/var/log/iptables.log"); };
destination d_acpid { file("/var/log/acpid.log"); };
destination d_console { usertty("root"); };

# Log everything to tty12
destination console_all { file("/dev/tty12"); };

filter f_auth { facility(auth); };
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { program(syslog-ng); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kernel { facility(kern) and not filter(f_iptables); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };
filter f_ppp { facility(local2); };
filter f_debug { not facility(auth, authpriv, news, mail); };
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news, cron) and not program(syslog-ng) and not filter(f_iptables); };
filter f_everything { level(debug..emerg) and not facility(auth, authpriv); };
filter f_emergency { level(emerg); };
filter f_info { level(info); };
filter f_notice { level(notice); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };
filter f_iptables { match("IN=" value("MESSAGE")) and match("OUT=" value("MESSAGE")); };
filter f_acpid { program("acpid"); };

log { source(src); filter(f_acpid); destination(d_acpid); };
log { source(src); filter(f_authpriv); destination(d_authlog); };
log { source(src); filter(f_syslog); destination(d_syslog); };
log { source(src); filter(f_cron); destination(d_cron); };
log { source(src); filter(f_daemon); destination(d_daemon); };
log { source(src); filter(f_kernel); destination(d_kernel); };
log { source(src); filter(f_lpr); destination(d_lpr); };
log { source(src); filter(f_mail); destination(d_mail); };
log { source(src); filter(f_news); destination(d_news); };
log { source(src); filter(f_ppp); destination(d_ppp); };
log { source(src); filter(f_user); destination(d_user); };
log { source(src); filter(f_uucp); destination(d_uucp); };
#log { source(src); filter(f_debug); destination(d_debug); };
log { source(src); filter(f_messages); destination(d_messages); };
log { source(src); filter(f_err); destination(d_errors); };
log { source(src); filter(f_emergency); destination(d_console); };
log { source(src); filter(f_everything); destination(d_everything); };
log { source(src); filter(f_iptables); destination(d_iptables); };

# Log everything to tty12
#log { source(src); destination(console_all); };

Last edited by lb.laboon (2014-12-09 21:29:53)

ihrwein · 2014-12-10 08:22:12

Could you please open an issue at GitHub? The developers cannot watch every forum so that might help a lot.

Darkimmortal · 2014-12-11 17:55:44

For me the high CPU usage stopped once syslog-ng had output the entire journal (including entries from several days ago) into the /var/log files. This took a few hours.

lb.laboon · 2014-12-11 19:07:54

It looks like that's the case for me as well. After reading your post I decided to reinstall the latest version and let it run for an hour or two and it appears to be stable now.

I am curious though as to why it suddenly started happening when it did. I'll keep an eye out to see if it starts acting up again tonight. If everything is still looking good tomorrow then I'll change this thread to solved.

Last edited by lb.laboon (2014-12-11 19:16:04)

lb.laboon · 2014-12-12 16:20:23

Didn't notice anything unusual last night, so I'm marking this solved.

Solution: Just let it run for a few hours.

Arch Linux

#1 2014-12-08 14:40:35

[SOLVED] syslog-ng using 100% CPU

#2 2014-12-08 15:33:57

Re: [SOLVED] syslog-ng using 100% CPU

#3 2014-12-08 15:39:14

Re: [SOLVED] syslog-ng using 100% CPU

#4 2014-12-08 16:21:45

Re: [SOLVED] syslog-ng using 100% CPU

#5 2014-12-08 19:03:36

Re: [SOLVED] syslog-ng using 100% CPU

#6 2014-12-08 21:44:59

Re: [SOLVED] syslog-ng using 100% CPU

#7 2014-12-09 10:47:00

Re: [SOLVED] syslog-ng using 100% CPU

#8 2014-12-09 19:10:53

Re: [SOLVED] syslog-ng using 100% CPU

#9 2014-12-10 08:22:12

Re: [SOLVED] syslog-ng using 100% CPU

#10 2014-12-11 17:55:44

Re: [SOLVED] syslog-ng using 100% CPU

#11 2014-12-11 19:07:54

Re: [SOLVED] syslog-ng using 100% CPU

#12 2014-12-12 16:20:23

Re: [SOLVED] syslog-ng using 100% CPU

Board footer