[systemd] SystemMaxUse option is not respected.

birdflesh · 2012-10-04 10:48:01

I'm a happy overall systemd user. My only concern is that permanent logs (saved in /var/log/journal) are growing beyond the set limit (50M) in /etc/systemd/journald.conf

SystemMaxUse=50M is the only change from the original file I made yet 'journalctl --disk-usage' returns 'Journals take up 58.4M on disk.'

Any thoughts on this?

Also, a minor problem, systemd pollutes my /var/tmp with a new systemd-private-XXXXXX folder on every boot. Does this happen to other systemd users?

Thank you for your answers

Last edited by birdflesh (2012-11-25 20:56:32)

falconindy · 2012-10-04 12:10:15

birdflesh wrote:

I'm a happy overall systemd user. My only concern is that permanent logs (saved in /var/log/journal) are growing beyond the set limit (50M) in /etc/systemd/journald.conf

It's an off by one error, so to speak (which I reported a while ago). It'll be rotated once you read SystemMaxUse+SystemMaxFileSize.

birdflesh wrote:

Also, a minor problem, systemd pollutes my /var/tmp with a new systemd-private-XXXXXX folder on every boot. Does this happen to other systemd users?

These are created for programs which set PrivateTmp=yes in their Service configuration.

berbae · 2012-10-04 12:59:35

falconindy wrote:

These are created for programs which set PrivateTmp=yes in their Service configuration.

I use ntpd.service and I see that these folders are not removed at shutdown. Why? And how to automatically clean /var/tmp of these remaining useless folders?

falconindy · 2012-10-04 20:23:14

Tmpfiles can clear /var/tmp for you.

teateawhy · 2012-10-04 21:14:21

I am also interested in whether it is reasonable to delete the folders in /var/tmp at shutdown or not.

Last edited by teateawhy (2012-10-04 21:15:29)

ninian · 2012-10-04 21:32:02

falconindy wrote:

Tmpfiles can clear /var/tmp for you.

Creating a file /etc/tmpfiles.d/remove-private-tmp.conf containing:

r /var/tmp/systemd-private-* - - - - -

seems to work for me.

birdflesh · 2012-10-05 08:53:16

Thank you all for your answers.

tmpfiles.d did the trick, would it be sane to have a configuration file like the one described by ninian added in the ntpd package?

On the journal size issue, I followed falconindy's advise but that didn't help. Total journal size again exceeded the sum of SystemMaxUse (50M) and SystemMaxFileSize (10M).

berbae · 2012-10-05 12:20:52

birdflesh wrote:

tmpfiles.d did the trick

Please can you explain what you did exactly to solve the issue with remaining folders in /var/tmp.
Can you post the conf file you created there?

Edit: Sorry I didn't read well it is already posted in ninian's post.

Last edited by berbae (2012-10-05 12:34:27)

cfr · 2012-10-05 23:25:06

Maybe I've misunderstood, but won't that configuration file destroy all /var/tmp/systemd-private-*? If something needs such a directory, won't this mess things up? That is, won't this remove the directories at boot rather than shutdown, say?

Last edited by cfr (2012-10-05 23:25:25)

birdflesh · 2012-10-06 15:19:37

There is a new thread about the temporary directories issue. Discussion about this can continue there.

I would very much appreciate some confirmation/more info about the issue described on topic.

birdflesh · 2012-10-09 15:25:36

OK, I removed and created /var/log/journal again, set a limit of 10M in SystemMaxUse and now logs rotate and never get any bigger in size.
I have no idea what could be the problem before but I can mark this as [solved].

berbae · 2012-10-25 08:52:26

birdflesh wrote:

OK, I removed and created /var/log/journal again, set a limit of 10M in SystemMaxUse and now logs rotate and never get any bigger in size.

Please, can you confirm that the output of 'journalctl --disk-usage' is always under the value of SystemMaxUse?
If so can you post your '/etc/systemd/journald.conf' file; and exactly what you have done and in what order, please.

I have not re-created the /var/log/journal directory yet, because I'm not sure if doing that when the systemd-journald.service is running is safe and clean.

Presently I get:

$ journalctl -b --no-pager _SYSTEMD_UNIT=systemd-journald.service
Logs begin at Sun, 30 Sep 2012 10:40:19 +0200, end at Sun, 30 Sep 2012 23:41:23 +0200.
Oct 25 09:58:45 arch64 systemd-journal[120]: Allowing runtime journal files to grow to 191.4M.
Oct 25 09:58:45 arch64 systemd-journal[120]: Journal started
Oct 25 09:58:51 arch64 systemd-journal[120]: Allowing system journal files to grow to 16.0M.

and

$ journalctl --disk-usage
Journals take up 22.3M on disk.

So the value is greater than 16M, even greater than 16 + 16/8 = 18M
my '/etc/systemd/journald.conf' is like that:

$ grep -v '^#' /etc/systemd/journald.conf                                                                             

[Journal]
SystemMaxUse=16M
SystemKeepFree=8M
ForwardToSyslog=no

After changing the values in it I ran 'systemctl daemon-reload', and the machine reboots each day.

I don't like the idea to re-create '/var/log/journal' while systemd-journald.service is running.
This doesn't seem very clean to me.

birdflesh · 2012-10-25 13:20:19

It's exactly my problem. As I said, it went away by recreating /var/log/journal.
My configuration has nothing special, just SystemMaxUse is set to 20M.
If you feel uncomfortable just deleting the folder, you can try stopping systemd-journald.service first.

berbae · 2012-10-25 15:32:46

Thanks birdflesh to answer my post; I am pleased it works for you, but personally I need more precise infos.
So for comparison, please would you mind posting your outputs of the same commands I used in my post #12.

I think it is important to clarify that matter of disk usage for the systemd-journald service and to control that securely.

birdflesh · 2012-10-25 17:39:25

$ journalctl -b --no-pager _SYSTEMD_UNIT=systemd-journald.service
-- Logs begin at Mon, 2012-10-22 09:23:23 EEST, end at Mon, 2012-10-22 14:18:53 EEST. --
Oct 25 20:27:45 birdflesh systemd-journal[123]: Allowing runtime journal files to grow to 192.8M.
Oct 25 20:27:45 birdflesh systemd-journal[123]: Journal started
Oct 25 20:27:47 birdflesh systemd-journal[123]: Allowing system journal files to grow to 20.0M.

$ journalctl --disk-usage
Journals take up 20.7M on disk.

$ grep -v '^#' /etc/systemd/journald.conf                                                                             
[Journal]
SystemMaxUse=20M
ForwardToSyslog=no

Please try running 'journalctrl --verify'. This will verify that files in your /var/log/journal are not corrupted.
I didn't try this when I had this problem so I'm not at all sure that it has something to do with it but try it anyway.

Last edited by birdflesh (2012-10-25 17:40:03)

berbae · 2012-10-26 08:02:24

@birdflesh: I see that your disk usage is greater than 20M; did you check to what value it grows up before going back to a lower than 20M?

birdflesh · 2012-10-26 08:12:57

Definitely no more than 20+20/8=22.5 .
So, have you tried to verify the integrity of your journal files?

berbae · 2012-10-26 15:20:29

Yes I ran the 'journalctl --verify' command and no errors were reported.

I decided to experiment the re-creation of /var/log/journal, from your testimony that it works.
I ran successively under root:

# systemctl stop systemd-journald.service
# cd /var/log
# mv journal/ journal.bak
# mkdir journal
# systemctl start systemd-journald.service

Nothing was re-created in /var/log/journal and there was nothing written in a new journal.
Not very reassuring at first.

Then I saw that a new upgrade of systemd was released, and I updated my system:

# pacman -Syu
# reboot

After that, a new journal was re-created. Nice to see it!

I now have to wait to confirm that the disk usage doesn't grow higher than 16 + 16/8 = 18M
This should take several days and I will post again when I have the data available.

Thanks birdflesh.

berbae · 2012-10-26 15:58:06

By the way, backups of old journal files can still be read by, for example:

journalctl -D /var/log/journal.bak/

passing the folder where the backups are kept.

berbae · 2012-11-24 16:29:18

I can now say that the birdflesh's trick of recreating /var/log/journal doesn't work for me:

$ journalctl --disk-usage
Journals take up 21.3M on disk.

when it should not have grown higher than 18M.

So how can I safely control the size of the systemd journal on the disk?

Or how can I safely clean /var/log/journal regularly?

This is an important issue to solve, as I cannot let the journal grows indefinitely.

As systemd is now used by more and more Arch users, has someone already solved that?

How the journal disk usage is controlled by other systemd users?

Thanks for giving suggestions.

cfr · 2012-11-25 00:38:59

I have the same problem - should be limited to 50M but stands at 171M. I don't know how to limit it...

However, I just ran verify and it found a problem with one file. What is the recommended procedure for dealing with this?

Last edited by cfr (2012-11-25 00:41:21)

berbae · 2012-11-25 10:50:35

@birdflesh: do you reboot your computer everyday or let it running permanently?
And, as at least two persons don't see that issue solved, would you mind please to delete the SOLVED tag from this thread tittle?

Other contributions are needed.

From 'man journald.conf':

SystemMaxFileSize= and RuntimeMaxFileSize= control how large individual journal files
may grow at maximum. This influences the granularity in which disk space is made available through rotation, i.e.
deletion of historic data. Defaults to one eighth of the values configured with SystemMaxUse= and RuntimeMaxUse=, so that usually seven rotated journal files are kept as history.

So it is intended to delete old journal files regularly, but maybe if the system is continuously running, I don't know.
Because on my system, more than 7 journal files are present and seemingly never deleted; but I shutdown/reboot my machine everyday.
The journal file of the preceding day is always rotated at the start of the machine, whatever its size, and no size control of the total disk usage is made at that time, and never is after that.
All this is at least not very clear, and requires some clarifications.

Is it intended that the machine should be running permanently?

And how to force a cleaning of old journal files in case of a daily shutdown/reboot?

Last edited by berbae (2012-11-25 10:51:39)

birdflesh · 2012-11-25 21:00:07

I removed "solved" from the title as it seems that this is still an issue for some people.

@barbae I usually reboot many times every day.

berbae · 2012-11-25 22:21:56

Thanks birdflesh to re-open this thread.
Can you confirm that you have always only 7 rotated journal files left in the journal directory?
On my machine a new journal file is created at each reboot but no old rotated journal files are ever deleted, even when the disk usage is much higher than SystemMaxUse (+1/8).
(BTW I'm just curious: why do you reboot so many times every day?)

cfr · 2012-11-25 23:30:24

I don't reboot every day. (Sometimes I reboot more than once in a day if I need to but often I'll go a few days without rebooting.) I sleep several times a day, though. (Actually, I sleep once but my laptop sleeps several.)

@berbae,
Do you see journal file corruption at all? I'm wondering if that is blocking the constraints in my case (but I don't know what to do about it yet).

Arch Linux

#1 2012-10-04 10:48:01

[systemd] SystemMaxUse option is not respected.

#2 2012-10-04 12:10:15

Re: [systemd] SystemMaxUse option is not respected.

#3 2012-10-04 12:59:35

Re: [systemd] SystemMaxUse option is not respected.

#4 2012-10-04 20:23:14

Re: [systemd] SystemMaxUse option is not respected.

#5 2012-10-04 21:14:21

Re: [systemd] SystemMaxUse option is not respected.

#6 2012-10-04 21:32:02

Re: [systemd] SystemMaxUse option is not respected.

#7 2012-10-05 08:53:16

Re: [systemd] SystemMaxUse option is not respected.

#8 2012-10-05 12:20:52

Re: [systemd] SystemMaxUse option is not respected.

#9 2012-10-05 23:25:06

Re: [systemd] SystemMaxUse option is not respected.

#10 2012-10-06 15:19:37

Re: [systemd] SystemMaxUse option is not respected.

#11 2012-10-09 15:25:36

Re: [systemd] SystemMaxUse option is not respected.

#12 2012-10-25 08:52:26

Re: [systemd] SystemMaxUse option is not respected.

#13 2012-10-25 13:20:19

Re: [systemd] SystemMaxUse option is not respected.

#14 2012-10-25 15:32:46

Re: [systemd] SystemMaxUse option is not respected.

#15 2012-10-25 17:39:25

Re: [systemd] SystemMaxUse option is not respected.

#16 2012-10-26 08:02:24

Re: [systemd] SystemMaxUse option is not respected.

#17 2012-10-26 08:12:57

Re: [systemd] SystemMaxUse option is not respected.

#18 2012-10-26 15:20:29

Re: [systemd] SystemMaxUse option is not respected.

#19 2012-10-26 15:58:06

Re: [systemd] SystemMaxUse option is not respected.

#20 2012-11-24 16:29:18

Re: [systemd] SystemMaxUse option is not respected.

#21 2012-11-25 00:38:59

Re: [systemd] SystemMaxUse option is not respected.

#22 2012-11-25 10:50:35

Re: [systemd] SystemMaxUse option is not respected.

#23 2012-11-25 21:00:07

Re: [systemd] SystemMaxUse option is not respected.

#24 2012-11-25 22:21:56

Re: [systemd] SystemMaxUse option is not respected.

#25 2012-11-25 23:30:24

Re: [systemd] SystemMaxUse option is not respected.

Board footer