You are not logged in.

#1 2013-02-21 19:17:21

Cromulent
Member
From: United Kingdom
Registered: 2011-08-28
Posts: 52

Is it safe to delete /var/log/journal log files?

My /var partition is starting to fill up slowly and 2.7GB of space is being taken up by the /var/log/journal directory. Is it safe to delete these files?

Offline

#2 2013-02-21 19:51:59

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: Is it safe to delete /var/log/journal log files?

Yes, it is.  As long as you don't need the logs for any debugging, it is perfectly okay.  You should take a look at /etc/systemd/journald.conf.  There is a setting to limit the amount of journals the system stores, so old ones get rotated out.  This setting isn't perfect though, as corrupted files apparently do not get counted against this total (at least this was the case a couple months ago, not sure if it is fixed now), but it will certainly help to keep from having your system's /var fill up like you are experiencing.

Offline

#3 2013-02-21 20:42:53

olive
Member
From: Belgium
Registered: 2008-06-22
Posts: 1,054

Re: Is it safe to delete /var/log/journal log files?

Cromulent wrote:

My /var partition is starting to fill up slowly and 2.7GB of space is being taken up by the /var/log/journal directory. Is it safe to delete these files?

You can delete all the files in /var/log/journal but do not delete the /var/log/journal directory itself.

Last edited by olive (2013-02-21 20:43:13)

Offline

#4 2013-02-22 06:52:49

Cromulent
Member
From: United Kingdom
Registered: 2011-08-28
Posts: 52

Re: Is it safe to delete /var/log/journal log files?

Thanks for the information. Saves me having to expand the size of the partition for a while.

Offline

#5 2013-02-22 11:06:17

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,080

Re: Is it safe to delete /var/log/journal log files?

It doesn't seem normal that nothing is deleted in your /var/log/journal/.../ directory.
Do you see corrupted journal files in it: those with a .journal~ suffix?

On my machine, only one month of entries are kept; I have a small value (16M) in the SystemMaxUse parameter, but it doesn't seem to be respected:

$ journalctl --disk-usage
Journals take up 20.5M on disk.

But I see that every rotated journal files with entries older than one month are deleted automatically; so the disk usage doesn't grow indefinitely.

$ grep -v '^#' /etc/systemd/journald.conf

[Journal]
SystemMaxUse=16M
ForwardToSyslog=no

It's not clear to me if the MaxFileSec parameter, which defaults to one month, is the explanation for the deletion of older than one month rotated journal files.

As I and others have observed, the cleaning of the /var/log/journal/.../ directory has some bugs not yet presently corrected.

I improved the thing when I added the 'shutdown' hook to the initial ramdisk created by mkinitcpio, but that doesn't solve every problem with the journald cleaning procedure.

Last edited by berbae (2013-02-22 15:18:00)

Online

#6 2013-02-22 15:37:46

PaulBx1
Member
Registered: 2008-10-18
Posts: 142

Re: Is it safe to delete /var/log/journal log files?

My log is full of .journal~ files. Great, just what I need, a non-functional log.

Offline

#7 2013-02-22 15:45:37

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: Is it safe to delete /var/log/journal log files?

I just delete em after a while... if my system didn't do it on its own.

Offline

#8 2013-02-22 22:30:11

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,080

Re: Is it safe to delete /var/log/journal log files?

Before manually removing the old rotated journal files in the /var/log/journal/.../ directory, a forced rotation can be tried with:

# systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service

which should also launch the cleanup procedure.

See 'man systemd-journald.service'.

In last resort, there is also the option of the 'volatile' Storage parameter of the systemd journal, with the syslog-ng daemon to store the logs on the disk in the traditional way, with the rotation/cleanup by logrotate.

But apart from the cleanup issues, the systemd journald log service seems fully functional, with a possible recourse to the traditional syslog-ng daemon.

Online

#9 2013-02-22 22:44:38

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: Is it safe to delete /var/log/journal log files?

Wow, I like that systemctl command above berbae! I have just been sending the SIGUSR2 via kill(all). Somehow your method above just appears much more correct.

Offline

#10 2013-04-09 09:15:46

farmerdave
Member
Registered: 2011-11-23
Posts: 66

Re: Is it safe to delete /var/log/journal log files?

PaulBx1 wrote:

My log is full of .journal~ files. Great, just what I need, a non-functional log.

I get this too, would really like to know why they appear as hidden instead of .journal like others seem to.

Offline

#11 2013-04-09 10:03:55

WorMzy
Forum Moderator
From: England
Registered: 2010-06-16
Posts: 4,882

Re: Is it safe to delete /var/log/journal log files?

A tilde at the end of a name is used to make files hidden, just like a dot at the start of a name. Traditionally, .$file has been for configs, and $file~ for backups.


Sakura:-
Mobo: ASUS P8Z77-V PRO // Processor: Intel Core i7-3770K 3.4GHz // GFX: nVidia GeForce GTX 970 Ti // RAM: 32GB (4x 8GB) Corsair DDR3 (@ 2133MHz) // Storage: 1x 3TB Seagate SATAII 5x 1TB Samsung SATAII, 2x 120GB Corsair SSD

Online

#12 2013-04-09 10:26:21

farmerdave
Member
Registered: 2011-11-23
Posts: 66

Re: Is it safe to delete /var/log/journal log files?

WorMzy wrote:

A tilde at the end of a name is used to make files hidden, just like a dot at the start of a name. Traditionally, .$file has been for configs, and $file~ for backups.

Yes, understand that. But I'm asking why I only get 1 .journal file, and have a list of over 50 backup files. Shouldn't I at least have a rotation of files that aren't hidden?

Offline

#13 2013-04-09 14:16:30

Thaodan
Member
From: Dortmund, Nordrein-Westfalen
Registered: 2012-04-28
Posts: 432

Re: Is it safe to delete /var/log/journal log files?

Its a bit OT but: is there a way to make the jounald logs smaller? I think about the disk usage compared to syslog-ng's text files that are mich smaler.


Linux odin 3.13.1-pf #1 SMP PREEMPT Wed Mar 5 21:47:28 CET 2014 x86_64 GNU/Linux

Offline

#14 2013-04-09 14:51:41

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,080

Re: Is it safe to delete /var/log/journal log files?

@farmerdave: If you have only one .journal and all others as .journal~, it may be that the journald service is not cleanly stopped.
In that case, you can try to add the 'shutdown' hook in the initramfs generated by mkinitcpio.
If you want more rotations, you need to change the value of 'SystemMaxUse' to a lower one in /etc/systemd/journald.conf
To force a rotation/cleanup, see my post #8.

@Thaodan see 'man journald.conf' to adjust the parameters to your liking, particularly 'SystemMaxUse' and 'SystemMaxFileSize'.

Online

#15 2013-04-09 14:53:49

Thaodan
Member
From: Dortmund, Nordrein-Westfalen
Registered: 2012-04-28
Posts: 432

Re: Is it safe to delete /var/log/journal log files?

berbae wrote:

@Thaodan see 'man journald.conf' to adjust the parameters to your liking, particularly 'SystemMaxUse' and 'SystemMaxFileSize'.

Yes I know but thoungt about making them smaller and not limiting their size. For example with disabling binary logging.


Linux odin 3.13.1-pf #1 SMP PREEMPT Wed Mar 5 21:47:28 CET 2014 x86_64 GNU/Linux

Offline

#16 2013-04-09 21:26:50

farmerdave
Member
Registered: 2011-11-23
Posts: 66

Re: Is it safe to delete /var/log/journal log files?

berbae wrote:

@farmerdave: If you have only one .journal and all others as .journal~, it may be that the journald service is not cleanly stopped.
In that case, you can try to add the 'shutdown' hook in the initramfs generated by mkinitcpio.
If you want more rotations, you need to change the value of 'SystemMaxUse' to a lower one in /etc/systemd/journald.conf
To force a rotation/cleanup, see my post #8.

Thanks, I'll look in to adding the 'shutdown' hook.

I have 3 computers, all with Arch on them and configured in much the same way. Especially when it comes to these core packages. My htpc is the only one with the singe journal file, and I have the ACPI wakeup/shutdown implemented. A likely source of the problem also? (The other two pc's have multiple files as well as the hidden backup files).

Offline

#17 2013-04-12 23:13:35

farmerdave
Member
Registered: 2011-11-23
Posts: 66

Re: Is it safe to delete /var/log/journal log files?

berbae wrote:

If you want more rotations, you need to change the value of 'SystemMaxUse' to a lower one in /etc/systemd/journald.conf
To force a rotation/cleanup, see my post #8.

I've set that parameter, but after 2 days my disk usage is greater than the value entered. Does the cleanup command need to be entered as a cronjob or should it happen automatically?

Offline

#18 2013-04-13 01:57:07

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,661

Re: Is it safe to delete /var/log/journal log files?

Is should be automatic.

This has actually been working for me recently so I assumed it was fixed. Just to say that the ~ files are not corrupt. They are just from unclean shutdowns, I think. (I have lots as my laptop was having issues. But I also get corrupt logs which are another issue.)

journalctl --verify

will show if you have corrupt logs.

This is my config file:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=10s
#RateLimitBurst=200
SystemMaxUse=50M
#SystemKeepFree=
SystemMaxFileSize=1M
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info

Every time I've checked journalctl --disk-usage recently, it has been just under 50M even though I've had corrupt logs, ~ logs and who knows what else in there. But perhaps that is because 50M is more than the minimum it will accept?

Last edited by cfr (2013-04-13 01:58:17)


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

#19 2013-04-13 03:56:34

farmerdave
Member
Registered: 2011-11-23
Posts: 66

Re: Is it safe to delete /var/log/journal log files?

Strange, the very first message in my current boot is

systemd-journal[128]: Allowing runtime journal files to grow to 796.9M.

But I have set SystemMaxUse=500M.

Offline

#20 2013-04-13 04:33:31

progandy
Member
Registered: 2012-05-17
Posts: 2,143

Re: Is it safe to delete /var/log/journal log files?

farmerdave wrote:

Strange, the very first message in my current boot is

systemd-journal[128]: Allowing runtime journal files to grow to 796.9M.

But I have set SystemMaxUse=500M.

I am not sure, but maybe it is like this:
- you specify 500M stored data
- journald calculates size for single file as 250M
- now there are always 2 files of 250M present while a new one is filled.
- When the new file is full, the oldest gets deleted and the journal rotates.

The difference from 750M to 796.2M could be the result of a buffer and maybe a different interpretation of the numbers (metric/decimal vs IEC/binary)

Offline

#21 2013-04-13 08:54:30

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,080

Re: Is it safe to delete /var/log/journal log files?

@farmerdave: 'runtime journal' is the volatile journal in ram; for the 'system journal' on disk, you should see a line like that:
'Allowing system journal files to grow to 500.0M.'
The rotation/cleanup of the journal files on disk should be automatic; there is one at startup if necessary, and after that, during runtime when it is required.
But if the journald service was uncleanly stopped, the rotation/cleanup at startup is seemingly hindered; and if that is so at every boot, the disk usage would not be well respected. In that case you can manually force a rotation/cleanup with the command in my post #8; there is no need to put that in a cron task; and that manual command should not even be necessary if everything works correctly.
Did you add the 'shutdown' hook in the initramfs? That's what fixed the uncleanly stopped journald service for me.

I precise that I speak from my personal observations; I am not a systemd developer.

Online

#22 2013-04-15 04:56:24

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: Is it safe to delete /var/log/journal log files?

Apparently the journal corruption issues have been addressed as of systemd 201.  So I have cleaned out my journal, leaving only the current system and user journals.  I checked those... and the system journal was corrupt.  So I rotated it out and removed the old ones... now we shall see if they remain uncorrupted.

I guess I hope I don't realize a sudden need for old logs any time soon. wink

Offline

#23 2015-10-11 22:28:40

joel.obrecht
Member
Registered: 2010-02-18
Posts: 9

Re: Is it safe to delete /var/log/journal log files?

Sorry to dig up this thread, but as I had the same problem, corrupted logs, ... I found that the most clean solution is this systemd command:

journalctl –vacuum-size=50M

Better than hand removing things or killing services...

Credit to: Link

So to keep this in memory until next time it happens to me I'd thought to post it here smile

Offline

Board footer

Powered by FluxBB