You are not logged in.

#1 2017-07-04 18:33:48

MountainX
Member
Registered: 2016-02-08
Posts: 371

systemd-journald not starting after boot up

I restored an Arch Linux system from a BTRFS snapshot. All is well except that journald has to be restarted after every boot:

    systemctl restart systemd-journald

If I don't do this, journalctl shows old output that ends with the last shutdown before the reboot.

Also, when checking the status of systemd units, I see messages similar to this:

     Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

I'm not even sure how to troubleshoot this. Suggestions?

Offline

#2 2017-07-05 14:43:41

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd-journald not starting after boot up

Run as root:

journalctl --disk-usage
journalctl --list-boots
journalctl --verify

Offline

#3 2017-07-05 17:15:58

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: systemd-journald not starting after boot up

This is all after running

systemctl restart systemd-journald

which I do after any reboot

# journalctl --disk-usage

Archived and active journals take up 1.5G in the file system.

# journalctl --list-boots

-20 f826495819014e378ee9e420c766363b Wed 2017-01-18 16:52:20 EST—Sun 2017-01-22 17:21:02 EST
-19 6ced29df5be84c5d884b99c7c3105b18 Mon 2017-01-23 20:02:41 EST—Tue 2017-01-24 03:09:08 EST
-18 7d5437e8c2cf44fa808ebf87145335a5 Wed 2017-01-25 17:57:44 EST—Sat 2017-01-28 21:00:22 EST
-17 d809fd0cfbbe49769093e32cc7384a8f Mon 2017-01-30 16:13:02 EST—Wed 2017-02-15 22:00:15 EST
-16 774dd0eea80c46b59321fcc18e14c5e1 Wed 2017-03-01 15:40:41 EST—Wed 2017-03-08 13:16:52 EST
-15 8700d58592e943049ba1ae2147c76954 Wed 2017-03-08 14:51:26 EST—Sat 2017-05-13 15:28:21 EDT
-14 f9b663578bc9434580e00208d8a2bba0 Sat 2017-05-13 16:06:57 EDT—Sun 2017-06-25 19:36:18 EDT
-13 9ede2fb876af4726bafd628736e2c6eb Tue 2017-06-27 19:17:59 EDT—Tue 2017-06-27 23:22:36 EDT
-12 56278359a9d54836afc0c806aaf8c446 Tue 2017-06-27 23:48:54 EDT—Wed 2017-06-28 00:08:19 EDT
-11 8549e0ef304d4f2aa535701836c8d18d Wed 2017-06-28 00:35:45 EDT—Wed 2017-06-28 01:45:42 EDT
-10 4e5119c5645f4b30a9a72f1444db6394 Wed 2017-06-28 01:48:00 EDT—Wed 2017-06-28 02:00:19 EDT
 -9 9c339ba5e549484a877b161b585e0048 Wed 2017-06-28 02:15:08 EDT—Wed 2017-06-28 02:36:23 EDT
 -8 483591860f7e496cb7641255b016cf2a Wed 2017-06-28 02:38:15 EDT—Wed 2017-06-28 22:51:31 EDT
 -7 b0de059def354b2cafbdeeff2461672f Sat 2017-07-01 05:32:50 EDT—Sun 2017-07-02 05:05:08 EDT
 -6 d357583e6b98473ea372b464d6d2d1b2 Sun 2017-07-02 06:47:36 EDT—Sun 2017-07-02 08:12:48 EDT
 -5 b61032ab2fbc450db12c8a647d5463ac Sun 2017-07-02 09:57:55 EDT—Sun 2017-07-02 11:22:20 EDT
 -4 f3e49639c1ba4c27819b673034f37857 Sun 2017-07-02 15:55:39 EDT—Mon 2017-07-03 09:52:03 EDT
 -3 16e27d7674c64b6e8a2354bc200bd4c6 Mon 2017-07-03 10:39:24 EDT—Mon 2017-07-03 10:45:18 EDT
 -2 fd21a712c51f41a983422f468526660a Tue 2017-07-04 01:35:09 EDT—Tue 2017-07-04 22:20:12 EDT
 -1 2305e53a41d24b3f9fa7bbaa47ae53a2 Tue 2017-07-04 22:32:24 EDT—Tue 2017-07-04 22:39:16 EDT
  0 7d3db1153fa04ab39e7cf8da72213225 Tue 2017-07-04 22:39:33 EDT—Wed 2017-07-05 13:10:45 EDT

# systemctl is-enabled systemd-journald.service

static

# journalctl --verify

# journalctl --verify
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@3c8e5145724e47098f20e8b0d04f9bbd-000000000085a196-000546656e5b5a3b.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@3c8e5145724e47098f20e8b0d04f9bbd-000000000087d4ef-0005467ae50481e8.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@3c8e5145724e47098f20e8b0d04f9bbd-00000000008a14e1-000546917c9b041e.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@bea411116a704de0a8cb14306ad8f893-0000000000000001-000546b771a1ceb7.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-00000000008d807e-000546b64c507c5a.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@04c446e2d4f148f6bf48877ee21fa426-0000000000000001-000546d2a294afcd.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-0000000000908060-000546d2a15697df.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@04c446e2d4f148f6bf48877ee21fa426-0000000000924cec-000546f329213aeb.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-000000000092c683-000546f74bbe7a9f.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-0000000000000001-000547322bb9df57.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-00000000009573e7-0005472f1ad3c7cf.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-00000000009901bd-0005475647e42e11.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1001@2206094064d84afd89cd13b57ab23da1-00000000009b928c-00054761d56673a5.journal              
7558088: Unused data (entry_offset==0)                                                                                                                    
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-00000000009c64bd-00054763ede73393.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-00000000009ec03a-000547ae3acb8f78.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-00000000009f1462-000547c744188f65.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1001@2206094064d84afd89cd13b57ab23da1-0000000000a055a4-000548344969bdca.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-00000000009f2f07-000547ced54042fb.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-0000000000a29fac-000549d2b3264bba.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-0000000000a300ce-00054a03cdace2e1.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-0000000000a50d09-00054aee1e9eb782.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-0000000000a52682-00054af79baad22f.journal              
7fffe50: Unused data (entry_offset==0)                                                                                                                    
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-0000000000a62831-00054b9787446e76.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-0000000000a63adc-00054bab87645705.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1001@2206094064d84afd89cd13b57ab23da1-0000000000a6282b-00054b978730d595.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-0000000000a8b08a-00054db52498c9d1.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-0000000000a91909-00054df820fe34c8.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1001.journal                                                                                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@698d2e31cae948669d177e9bbe44454c-0000000000ab1d3c-00054f6f5298aab5.journal                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@9541996ffc264dd3bcc2f6506d88fd7d-0000000000ab8cfe-00054fdabee317ae.journal              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@000552fb8f7c1f56-a5d8f6082b76bef9.journal~                                                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@0005535360eabf9d-e233853da7c1c7f3.journal~                                                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@0005535b726bd73c-5a56eca7aeff7e93.journal~                                              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@0005535b7675b8bd-08c7a36078251ecf.journal~                                                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@0005535c4c089193-bd7854b565d77e58.journal~                                                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@0005535c4dbf3d00-0314cfd9736237cd.journal~                                              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system@0005537a1c1fb347-2143f48254b77728.journal~                                                 
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/system.journal                                                                                    
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000@0005537a257618f6-57df31a3e3a79602.journal~                                              
PASS: /var/log/journal/1b2906f6058b4899a2c9f55ffeba200a/user-1000.journal  

Last edited by MountainX (2017-07-06 03:41:24)

Offline

#4 2017-07-05 18:01:24

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: systemd-journald not starting after boot up

Would the suggestions here help?

https://jonarcher.info/2015/08/ensure-s … n-failure/

Restart=always
RestartSec=3

Offline

#5 2017-07-06 08:20:42

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd-journald not starting after boot up

This is already in the systemd-journald.service file:

Restart=always
RestartSec=0

After a reboot and before you restart systemd-journald, what is the output of:

systemctl status systemd-journald.service

Then after the restart of the journal, post a link to the output of:

journalctl -b

and maybe also:

journalctl -b -1

to see if the shutdown process shows errors.

Offline

#6 2017-07-06 20:12:01

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: systemd-journald not starting after boot up

berbae wrote:

This is already in the systemd-journald.service file:

Restart=always
RestartSec=0

After a reboot and before you restart systemd-journald, what is the output of:

systemctl status systemd-journald.service

Then after the restart of the journal, post a link to the output of:

journalctl -b

and maybe also:

journalctl -b -1

to see if the shutdown process shows errors.

I will follow up with all that. The machine is used daily so I can only work on it after hours. However, if I recall correctly from prior testing, the status of systemd-journald service is that it is started. It does not show any errors. Nor are there relevant errors in journalctl. The only symptom is that journalctl does not show any entries after the last shutdown until I restart systemd-journald service.

Offline

#7 2017-07-07 13:38:50

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd-journald not starting after boot up

I'd prefer seeing the real output and log messages rather than your descriptions by memory.
Thanks for providing what I asked for before.

But maybe the journal did not flush its content to the disk yet when you try to see it.
You should have something like this:

systemctl --no-pager status systemd-journald.service                                                                                                                      
● systemd-journald.service - Journal Service                                                                                                                                                 
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)                                                                                        
   Active: active (running) since Fri 2017-07-07 15:06:06 CEST; 26min ago                                                                                                                    
     Docs: man:systemd-journald.service(8)                                                                                                                                                   
           man:journald.conf(5)                                                                                                                                                              
 Main PID: 176 (systemd-journal)                                                                                                                                                             
   Status: "Processing requests..."                                                                                                                                                          
    Tasks: 1 (limit: 4915)                                                                                                                                                                   
   CGroup: /system.slice/systemd-journald.service                                                                                                                                            
           └─176 /usr/lib/systemd/systemd-journald                                                                                                                                           
                                                                                                                                                                                             
juil. 07 15:06:06 arch64 systemd-journald[176]: Journal started                                                                                                                              
juil. 07 15:06:06 arch64 systemd-journald[176]: Runtime journal (/run/log/journal/2f5647fc01c341ba8aab650be63265da) is 8.0M, max 190.9M, 182.9M free.
juil. 07 15:06:06 arch64 systemd-journald[176]: Time spent on flushing to /var is 77.802ms for 753 entries.
juil. 07 15:06:06 arch64 systemd-journald[176]: System journal (/var/log/journal/2f5647fc01c341ba8aab650be63265da) is 112.0M, max 100.0M, 0B free.

Of course if /var is on a separate partition it must be mounted.

Offline

#8 2017-09-12 05:23:05

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: systemd-journald not starting after boot up

Here are the logs requested. I only now rebooted the system since July!
I hope you are still interested in this issue :-)

https://gist.github.com/anonymous/726e9 … fd7f3032a2

The first thing I see of interest is the shutdown message:

Sep 11 15:59:39 box2 systemd[1]: Unmounting /var/log...
Sep 11 15:59:39 box2 systemd[1]: var-log.mount: Mount process exited, code=exited status=32
Sep 11 15:59:39 box2 systemd[1]: Failed unmounting /var/log. 

I have /var/log in a separate btrfs volume because of the filesystem layout I use for snapper. It was inspired by:
Snapper/BTRFS layout for easily restoring files, or entire system / Installation / Arch Linux Forums
https://bbs.archlinux.org/viewtopic.php?id=194491

However, I have other systems using this same layout and they do not seem to have the same problem with journal output. However, on this system, after this reboot the journal output was not truncated as usual.  Referring to the message "Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable" maybe the journal output is incomplete to varying degrees. I have seen it be lacking all output since the reboot. But this time, I did not see anything missing.

Offline

Board footer

Powered by FluxBB