Failed to start Enable Persistent Storage in systemd-network

phunni · 2024-09-14 18:55:00

I get this message during every boot. It doesn't prevent booting, but I would like to fix it.

I ran "journalctl | grep systemd-networkd" with the results:

Sep 14 19:44:14 fjola.monkeypower.co.uk systemd-networkd[859]: lo: Link UP
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd-networkd[859]: lo: Gained carrier
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd-networkd[859]: Enumeration completed
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd[1]: Starting Enable Persistent Storage in systemd-networkd...
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd-networkd[859]: The persistent storage is on read-only filesystem.
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd[1]: systemd-networkd-persistent-storage.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd[1]: systemd-networkd-persistent-storage.service: Failed with result 'exit-code'.
Sep 14 19:44:14 fjola.monkeypower.co.uk systemd[1]: Failed to start Enable Persistent Storage in systemd-networkd.
Sep 14 19:44:15 fjola.monkeypower.co.uk systemd-networkd[859]: eno1: Link UP
Sep 14 19:44:18 fjola.monkeypower.co.uk systemd-networkd[859]: eno1: Gained carrier
Sep 14 19:44:19 fjola.monkeypower.co.uk systemd-networkd[859]: eno1: Gained IPv6LL
Sep 14 19:46:14 fjola.monkeypower.co.uk systemd-networkd-wait-online[879]: Timeout occurred while waiting for network connectivity.
Sep 14 19:46:14 fjola.monkeypower.co.uk systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 19:46:14 fjola.monkeypower.co.uk systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.

I notice two interesting things here:

The persistent storage is on read-only filesystem.

and also the fact that en01 only comes up after the failure.

Part of the problem is that I'm not entirely clear which service is causing this or how to make it wait for the network - if that's the cause of the problem.

As for the read-only filesystem - that's certainly possible as I run my system as an immutable and stateless system. It's not, however, at all clear from the error message exactly which filesystem it's trying to use as persistent storage.

Any suggestions gratefully received - thank in advance!

seth · 2024-09-14 21:51:03

/usr/lib/systemd/system/systemd-networkd-persistent-storage.service runs "networkctl persistent-storage yes"
https://github.com/systemd/systemd/pull … 19f0cda28c had it "RequiresMountsFor=/var/lib/systemd/network" so I'd look there though afaict the requirement is gone from the service.

yxcv · 2024-09-14 21:56:34

whois fjola.monkeypower.co.uk
Error for "fjola.monkeypower.co.uk".
This domain cannot be registered because it contravenes the Nominet UK
naming rules. The reason is:
the domain name contains too many parts.
WHOIS lookup made at 22:54:04 14-Sep-2024

.....

seth · 2024-09-14 22:03:43

whois www.google.co.uk                                                                                   
                                                                                                                                
    Error for "www.google.co.uk".                                                                                               
                                                                                                                                
    This domain cannot be registered because it contravenes the Nominet UK                                                      
    naming rules.  The reason is:                                                                                               
      the domain name contains too many parts.

And now type www.google.co.uk into a browser…
https://en.wikipedia.org/wiki/Hostname#Syntax which isn't even the same as a https://en.wikipedia.org/wiki/Domain_name (though it's good to not stretch it)

tl;dr The hostname is hardly relevant here.

phunni · 2024-09-20 14:06:40

seth wrote:

/usr/lib/systemd/system/systemd-networkd-persistent-storage.service runs "networkctl persistent-storage yes"
https://github.com/systemd/systemd/pull … 19f0cda28c had it "RequiresMountsFor=/var/lib/systemd/network" so I'd look there though afaict the requirement is gone from the service.

Thanks - that's a useful starting place.

Also - the domain name is hardly relevant, but it's all on a local intranet.

phunni · 2024-09-20 18:52:47

Well - I'm baffled. I tried adding a drop in to enable debugging, but there's no further output. /var/lib/systemd/network exists and is writeable...

Without clearly being able to see which read only files it's trying to access it's hard to figure this out.

seth · 2024-09-20 19:00:59

What kind of "debugging" exactly?
Knowing systemd, networkctl will likely just dbus-talk to some other process (in doubt systemd)
I'd probably check

findmnt -T /var/lib/systemd/network >> /tmp/systemd.sucks

as an ExecStartPre command to see the state of the FS right before the command runs.

phunni · 2024-09-28 14:46:04

What kind of "debugging" exactly?

This kind: https://wiki.archlinux.org/title/System … _a_service I made a drop-in as suggested there.

I'll give that ExecStartPre a go - thanks.

It's clearly failing due to some sort of attempt to access/mount or something a read only file system. That's not surprising - most of the system is read only. If only I could find out what it was trying to access and why I might be able to diagnose the problem...

seth · 2024-09-30 14:08:07

You could audit /var/lib/systemd/network, https://wiki.archlinux.org/title/Audit_ … ies_access but the tricky part will be to load it early.
Have you also tried re-adding ""RequiresMountsFor=/var/lib/systemd/network"" to the service?

Arch Linux

#1 2024-09-14 18:55:00

Failed to start Enable Persistent Storage in systemd-network

#2 2024-09-14 21:51:03

Re: Failed to start Enable Persistent Storage in systemd-network

#3 2024-09-14 21:56:34

Re: Failed to start Enable Persistent Storage in systemd-network

#4 2024-09-14 22:03:43

Re: Failed to start Enable Persistent Storage in systemd-network

#5 2024-09-20 14:06:40

Re: Failed to start Enable Persistent Storage in systemd-network

#6 2024-09-20 18:52:47

Re: Failed to start Enable Persistent Storage in systemd-network

#7 2024-09-20 19:00:59

Re: Failed to start Enable Persistent Storage in systemd-network

#8 2024-09-28 14:46:04

Re: Failed to start Enable Persistent Storage in systemd-network

#9 2024-09-30 14:08:07

Re: Failed to start Enable Persistent Storage in systemd-network

Board footer