Journald out of control

ruurdkrekt · 2012-10-15 08:51:41

Good morning,

Today I noticed that I have thousands of journald error lines in my log files.

Oct 15 09:40:03 localhost systemd-journald[10363]: Failed to write entry, ignoring: Cannot allocate memory

I did create the directory /var/log/journal/ which does have journal files in it.
So how come I am getting memory errors (around three per minute) if the journal is storing it's logs to this directory? Is there a way to solve this?
I am running systemd version 194.

Alternatively if this is for example a memory leak, is it possible to limit the amount of memory space used to a few MB and then bind syslog-ng with journal socket to store the logs?

ruurdkrekt · 2012-10-15 13:44:14

An additonal error that is now popping up is:

Oct 15 14:38:09 localhost systemd-journald[24677]: Failed to write entry, ignoring: Invalid argument

For some reason this changed around 12:00 without me making any changes to the system. So now it does not give the memory error anymore but instead it gives the above.
It still occurs regularly, between 2 to 8 times per minute. So as you can imagine my logs are flooded with this.

Also, if I check the status of systemd-journald in systemctl then it does not come up with these errors. These errors only popup in the /var/log/errors.log file.
If I connect to the console of my VPS then the console is flooded with these errors as well, so I am unable to use the console. However when I logon using SSH I do not notice any of it. I did put loglevel=3 in my kernel parameters as mentioned in the systemd wiki page.

does anybody have any ideas why this happens?

Morrad · 2012-10-15 21:59:51

It would be helpful to have a bit more information.

Do you have systemd set up to write out to log files through syslog-ng as specified in the wiki, or are you not keeping persistent logs?

When did you switch to using systemd? Did you do anything 'out of the ordinary' during that switch?

cfr · 2012-10-16 00:11:14

Morrad wrote:

Do you have systemd set up to write out to log files through syslog-ng as specified in the wiki, or are you not keeping persistent logs?

More information would definitely be helpful but this is a false dilemma. The OP said that s/he has created /var/log/journal. That should trigger persistent logs without syslog-ng. You only need the latter if you want to use the traditional system logging daemon rather than relying on systemd's journal facilities.

@OP,
How are you checking systemd journal? Are you examining the journal e.g. using journalctl or do you just mean that the journal service doesn't report an error? (That seems odd in any case, I admit.)

Last edited by cfr (2012-10-16 00:13:16)

ruurdkrekt · 2012-10-16 08:27:31

Hi Morrad & cfr, thank you for your reply.

I switched to a full systemd implementation around 4 weeks ago. I made the logs persistent by creating the directory in /var/log/journal/
I indeed used the syslog-ng daemon in combination with journald. Because I was not sure on how to use journalctl and did not want to loose anything so I figured better have it double then not at all, and learn how to use journalctl along the way.

As far as I knew everything worked fine, however when I accessed the console of my VPS (instead of going through SSH) it was filling up rapidly with these memory errors. Here I did get confused as I did put the loglevel=3 in my kernel paramters and according to the wiki things should not be logged to the console that way.

So next thing I did was check the status of journald by running:

[root@Yotta ruurd]# systemctl status systemd-journald
systemd-journald.service - Journal Service
          Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
          Active: active (running) since Mon, 15 Oct 2012 15:13:01 +0100; 18h ago
            Docs: man:systemd-journald.service(8)
                  man:journald.conf(5)
        Main PID: 31859 (systemd-journal)
          Status: "Processing requests..."
          CGroup: name=systemd:/system/systemd-journald.service
                  â 31859 /usr/lib/systemd/systemd-journald

This came up with that everything is running fine and also no sign of these memory errors appeared. As the systemctl status option normally also gives a few lines of log file.
However in /var/log/errors.log all the mentioned errors were available and there were thousands of them.

To stop these errors I changed the journald.conf:

[Journal]
Storage=none
ForwardToSyslog=yes
ForwardToKMsg=yes
ForwardToConsole=no

This did stop the errors to occur and also the console is less busy now, although still occasional logs are being written to it, which is still weird.

So the server where this happens on is a VPS, I also have a small download server at home with Arch on it and also switched to systemd on this machine. I used the same wiki and did everything exactly the same! This machine is running smooth and has no problems or errors whatsoever. Of course this machine is more lightweight and also does not run as much programs and websites as my VPS one.

Morrad · 2012-10-16 20:49:58

One thing you can check is the 'SystemMaxUse' in '/etc/system/systemd-journald.conf' to see how much memory journald has allocated.

ruurdkrekt · 2012-10-16 21:22:08

Morrad wrote:

One thing you can check is the 'SystemMaxUse' in '/etc/system/systemd-journald.conf' to see how much memory journald has allocated.

I am unable to find the file you are referring to, however in the /etc/systemd/journald.conf that option is commented out. Which I believe means that journald will use 10% of the available space?

doug piston · 2012-10-17 02:24:15

If I read the Wiki correctly if it is commented out it will be restricted to 10% of your root partition. If it is un-commented I think you'd need to specify a size or maybe it will go on forever? I am unclear on it but I would add a size limit.

Last edited by doug piston (2012-10-17 02:24:47)

Morrad · 2012-10-17 02:42:31

ruurdkrekt wrote:

I am unable to find the file you are referring to, however in the /etc/systemd/journald.conf that option is commented out. Which I believe means that journald will use 10% of the available space?

(Apologies, at work I am sadly afflicted by Windows, and found the wrong information on Google.)

That is indeed where that file is, and you are correct that the journal should be topping out at 10% of the available root or var partition.

I agree with doug, you should try adding a (large) size limit at least temporarily to see if it resolves what you are seeing. Also check to see how large your journal has become. You may now be seeing the consequence of another program filling up your allocated log space over the last few weeks.

ruurdkrekt · 2012-10-17 08:24:33

I currently have this journald.conf file, setting the Storage to none did stop all the error messages showing up:

[Journal]
Storage=none
ForwardToSyslog=yes
ForwardToKMsg=yes
ForwardToConsole=no

To try out if the errors stop if I set the SystemMaxUse variable, my log file now looks like this:

Storage=persistent
ForwardToSyslog=yes
ForwardToKMsg=yes
ForwardToConsole=no
SystemMaxUse=5G

Restarting systemd-journald resulted into my console being flooded and my /var/log/errors.log (Still running the syslog-ng daemon) having 49 errors in 15 seconds saying:

Oct 17 09:20:28 localhost systemd-journald[6550]: Failed to write entry, ignoring: Invalid argument

[root@Yotta ruurd]# systemctl status systemd-journald
systemd-journald.service - Journal Service
          Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
          Active: active (running) since Wed, 17 Oct 2012 09:19:13 +0100; 3min 57s ago
            Docs: man:systemd-journald.service(8)
                  man:journald.conf(5)
        Main PID: 6550 (systemd-journal)
          Status: "Processing requests..."
          CGroup: name=systemd:/system/systemd-journald.service
                  â 6550 /usr/lib/systemd/systemd-journald

Arch Linux

#1 2012-10-15 08:51:41

Journald out of control

#2 2012-10-15 13:44:14

Re: Journald out of control

#3 2012-10-15 21:59:51

Re: Journald out of control

#4 2012-10-16 00:11:14

Re: Journald out of control

#5 2012-10-16 08:27:31

Re: Journald out of control

#6 2012-10-16 20:49:58

Re: Journald out of control

#7 2012-10-16 21:22:08

Re: Journald out of control

#8 2012-10-17 02:24:15

Re: Journald out of control

#9 2012-10-17 02:42:31

Re: Journald out of control

#10 2012-10-17 08:24:33

Re: Journald out of control

Board footer