Journald stops logging unless manually restarted

rixz · 2017-01-24 11:08:40

This is a very strange issue. The daemon logs, but only if I restart it manually and the system reboots do not count toward this.

I do not understand why, nor I have any idea how to t-shoot.

Here's the details.

A few days ago I noticed that systemd-journald.service does not log anymore.
First thing I did I restarted the service manually (well maybe something got stuck - I said to myself), and it all went back to normal however on the next reboot I noticed that the service does not log again despite the reboot.

When I say it stops logging I do not mean that the service is offline, but that it simply does not log the messages.

Here's the journal output after I rebooted my PC.
After the reboot I immediately issued systemctl restart nginx to produce some logs, but there is nothing in the log until the last message from the shutdown.

Please note the nginx here is just an example service, there are a couple of other semi-production services here including a mail server. So all in all the system is OK, except that I have to manually issue systemctl status systemd-journald.service to the otherwise active jourlnald service to make it to log the messages.

~ # systemctl reboot

...
...
...


~ # systemctl restart nginx

~ # journalctl --lines 30
-- Logs begin at Wed 2016-09-21 17:53:04 CEST, end at Tue 2017-01-24 11:56:25 CET. --
Jan 24 11:55:40 kerrigan systemd[1]: Stopped target Swap.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivating swap /dev/disk/by-partuuid/ad140bb7-02...
Jan 24 11:55:41 kerrigan systemd[1]: Unmounted /home.
Jan 24 11:55:41 kerrigan systemd[1]: Unmounted /boot.
Jan 24 11:55:41 kerrigan systemd[1]: Stopped File System Check on /dev/disk/by-uuid/0063b985-4eca-4ee6-8650-a1922bf14bf1.
Jan 24 11:55:41 kerrigan systemd[1]: Stopped File System Check on /dev/disk/by-uuid/2d7e7d7b-4298-48f7-961f-203d1ad8f8a3.
Jan 24 11:55:41 kerrigan systemd[1]: Removed slice system-systemd\x2dfsck.slice.
Jan 24 11:55:41 kerrigan systemd[1]: Stopped target Local File Systems (Pre).
Jan 24 11:55:41 kerrigan systemd[1]: Stopped Create Static Device Nodes in /dev.
Jan 24 11:55:41 kerrigan systemd[1]: Stopped Remount Root and Kernel File Systems.
Jan 24 11:55:41 kerrigan systemd[1]: Reached target Shutdown.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/disk/by-uuid/94fd6152-36e1-4b47-be99-272bfad1decc.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part2.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/disk/by-partuuid/ad140bb7-02.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/sda2.
Jan 24 11:55:41 kerrigan systemd[1]: Reached target Unmount All Filesystems.
Jan 24 11:55:41 kerrigan systemd[1]: Reached target Final Step.
Jan 24 11:55:41 kerrigan systemd[1]: Starting Reboot...
Jan 24 11:55:41 kerrigan systemd[1]: Shutting down.
Jan 24 11:55:41 kerrigan kernel: systemd-shutdow: 37 output lines suppressed due to ratelimiting
Jan 24 11:55:41 kerrigan systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jan 24 11:55:41 kerrigan systemd-journald[4759]: Journal stopped

Here is some output to get a picture at the moment and after I issue a restart of the journald.service.

~ # systemctl status systemd-journald.service
* systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Tue 2017-01-24 11:56:00 CET; 3min 59s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 185 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 19660)
   CGroup: /system.slice/systemd-journald.service
           `-185 /usr/lib/systemd/systemd-journald

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


~ # systemctl restart systemd-journald.service

~ # systemctl restart nginx

~ # journalctl --lines 30
-- Logs begin at Wed 2016-09-21 17:53:04 CEST, end at Tue 2017-01-24 12:01:24 CET. --
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part2.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/disk/by-partuuid/ad140bb7-02.
Jan 24 11:55:41 kerrigan systemd[1]: Deactivated swap /dev/sda2.
Jan 24 11:55:41 kerrigan systemd[1]: Reached target Unmount All Filesystems.
Jan 24 11:55:41 kerrigan systemd[1]: Reached target Final Step.
Jan 24 11:55:41 kerrigan systemd[1]: Starting Reboot...
Jan 24 11:55:41 kerrigan systemd[1]: Shutting down.
Jan 24 11:55:41 kerrigan kernel: systemd-shutdow: 37 output lines suppressed due to ratelimiting
Jan 24 11:55:41 kerrigan systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jan 24 11:55:41 kerrigan systemd-journald[4759]: Journal stopped
-- Reboot --
Jan 24 12:01:20 kerrigan systemd-journald[185]: Received SIGTERM from PID 1 (systemd).
Jan 24 12:01:20 kerrigan systemd[1]: Stopped Journal Service.
Jan 24 12:01:20 kerrigan systemd[1]: Starting Journal Service...
Jan 24 12:01:20 kerrigan systemd-journald[716]: Journal started
Jan 24 12:01:20 kerrigan systemd-journald[716]: System journal (/var/log/journal/3da4339c488f4871ad85a999d9736ce8) is 152.0M, max 3.1G, 2.9G free.
Jan 24 12:01:20 kerrigan systemd[1]: Started Journal Service.
Jan 24 12:01:20 kerrigan systemd[1]: Starting Flush Journal to Persistent Storage...
Jan 24 12:01:20 kerrigan systemd[1]: Started Flush Journal to Persistent Storage.
Jan 24 12:01:24 kerrigan systemd[1]: Stopping A high performance web server and a reverse proxy server...
Jan 24 12:01:24 kerrigan systemd[1]: Stopped A high performance web server and a reverse proxy server.
Jan 24 12:01:24 kerrigan systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 24 12:01:24 kerrigan systemd[1]: Started A high performance web server and a reverse proxy server.

Last edited by rixz (2017-01-24 11:11:41)

rixz · 2017-01-25 12:31:31

I implemented a workaround by creating a systemd timer that will restart the journald daemon 1min after reboot and restart the daemon every 8 hours (just in case). Still, it is not a solution, it is a workaround.

Arch Linux

#1 2017-01-24 11:08:40

Journald stops logging unless manually restarted

#2 2017-01-25 12:31:31

Re: Journald stops logging unless manually restarted

Board footer