NFS mounts started to fail at boot

Galicarnax · 2023-04-02 11:52:07

I had a couple of NFS mounts defined in my fstab. They worked fine for years, but recently (maybe after some update), they fail to mount at boot. I still can mount them manually without problem. This is what I have in fstab for one such NFS mount:

servername:/exporteddir /mnt/shared nfs defaults,x-systemd.device-timeout=3,x-systemd.mount-timeout=5,retrans=5,_netdev 0 0

When after boot I check the status of those mounts with systmectl or journalctl, I can see that it failed because

mount.nfs: Network is unreachable

Following Arch wiki, I removed those entries from fstab and defined mount units instead. That didn't make any difference, even though I put network-online target as a dependency:

[Unit]
Description=Mount shared folder
Requires=network-online.target
After=network-online.target
[Mount]
What=servername:/exporteddir
Where=/mnt/shared
Type=nfs
Options=defaults,_netdev
[Install]
WantedBy=multi-user.target

Starting the units manually with systemctl start mnt-shared.mount does the job, but they fail at boot.
What could be happening here? Why the network-online dependency is ignored?

seth · 2023-04-02 12:40:49

Check the system journal, I rather suspect that the network-online.target is "reached" prematurely.
See eg. https://bbs.archlinux.org/viewtopic.php?id=284361

Galicarnax · 2023-04-03 13:15:16

I have checked the journal, but I am not sure I can definitely say from there I see the problem. The most suspicious thing seems to be that after systemd-networkd says the eno1 link is up, immediately after the kernel says eno1 link is down. Here I provide the relevant entries from the journal, after the boot had started:

...
Apr 02 17:37:57 zor systemd[1]: Starting Network Configuration...
...
Apr 02 17:37:57 zor systemd-networkd[293]: lo: Link UP
Apr 02 17:37:57 zor systemd-networkd[293]: lo: Gained carrier
Apr 02 17:37:57 zor systemd-networkd[293]: Enumeration completed
...
Apr 02 17:37:57 zor systemd[1]: Started Network Configuration.
Apr 02 17:37:57 zor systemd[1]: Reached target Network.
...
Apr 02 17:37:57 zor systemd[1]: Starting Wait for Network to be Configured...
...
Apr 02 17:37:57 zor systemd[1]: Reached target Local File Systems.
...
Apr 02 17:37:57 zor kernel: r8169 0000:03:00.0: enabling device (0000 -> 0003)
Apr 02 17:37:57 zor kernel: r8169 0000:03:00.0 eth0: RTL8168h/8111h, 54:05:db:be:95:ab, XID 541, IRQ 72
Apr 02 17:37:57 zor kernel: r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
...
Apr 02 17:37:57 zor systemd[1]: Finished Wait for Network to be Configured.
Apr 02 17:37:57 zor systemd[1]: Reached target Network is Online.
Apr 02 17:37:57 zor systemd[1]: Mounting Mount raspi at /mnt/raspi...
Apr 02 17:37:57 zor systemd[1]: Mounting Mount torrents...
Apr 02 17:37:57 zor systemd-networkd[293]: eth0: Interface name change detected, renamed to eno1.
...
Apr 02 17:37:57 zor systemd-networkd[293]: eno1: Configuring with /etc/systemd/network/10-eno1.network.
...
Apr 02 17:37:58 zor systemd-networkd[293]: eno1: Link UP
Apr 02 17:37:58 zor kernel: r8169 0000:03:00.0 eno1: Link is Down
...
Apr 02 17:37:58 zor systemd[1]: Starting Network Time Synchronization...
...
Apr 02 17:37:58 zor systemd[1]: Started Network Time Synchronization.
Apr 02 17:37:58 zor systemd[1]: Reached target System Time Set.
...
Apr 02 17:37:58 zor systemd[1]: Reached target Socket Units.
...
Apr 02 17:37:58 zor systemd[1]: Reached target Basic System.
...
Apr 02 17:37:58 zor mount[367]: mount.nfs: Network is unreachable
Apr 02 17:37:58 zor mount[366]: mount.nfs: Network is unreachable
...
Apr 02 17:37:58 zor systemd[1]: mnt-raspi.mount: Mount process exited, code=exited, status=32/n/a
Apr 02 17:37:58 zor systemd[1]: mnt-raspi.mount: Failed with result 'exit-code'.
Apr 02 17:37:58 zor systemd[1]: Failed to mount Mount raspi at /mnt/raspi.
Apr 02 17:37:58 zor systemd[1]: mnt-torrents.mount: Mount process exited, code=exited, status=32/n/a
Apr 02 17:37:58 zor systemd[1]: mnt-torrents.mount: Failed with result 'exit-code'.
...
Apr 02 17:37:58 zor systemd[1]: Failed to mount Mount torrents.
...

As indicated in the thread you referenced, I could create a separate service unit with sleep command. But that renders the network-online.target dependencies useless. I'd like to find the culprit.

Last edited by Galicarnax (2023-04-03 13:32:30)

seth · 2023-04-03 13:30:35

Please edit your post and use code tags, not quote tags.

the network-online.target *is* useless.
Depending on whether you use NM or systemd-networkd and your HW configuration it's more or less useless, but it is completely useless.
Welcome to lennarware.

I am not sure I can definitely say from there I see the problem

If you want a second opinion, please post your complete system journal for the boot:

sudo journalctl -b | curl -F 'file=@-' 0x0.st

The mounts seem to happen even before the silly NIC name change, https://wiki.archlinux.org/title/Networ … face_names

MatzeR · 2023-12-04 11:54:57

I have the same issue since a couple of weeks. My nfs mounts started to run into a timeout. When the system is fully up I can mount the filesystems on cmd line "mount -av" without any issues. I have posted my full boot journal with the above mentioned cmd. Here is small part of it.
____________________________________
Dez 04 12:34:04 idefix NetworkManager[621]: <info> [1701689644.1855] manager: startup complete
Dez 04 12:34:04 idefix systemd[1]: Finished Network Manager Wait Online.
Dez 04 12:34:04 idefix systemd[1]: Reached target Network is Online.
Dez 04 12:34:04 idefix systemd[1]: Mounting /media/Bibliothek...
Dez 04 12:34:04 idefix systemd[1]: Mounting /media/Musik...
Dez 04 12:34:04 idefix systemd[1]: Mounting /media/Videothek...
Dez 04 12:34:04 idefix kernel: FS-Cache: Loaded
Dez 04 12:34:04 idefix kernel: RPC: Registered named UNIX socket transport module.
Dez 04 12:34:04 idefix kernel: RPC: Registered udp transport module.
Dez 04 12:34:04 idefix kernel: RPC: Registered tcp transport module.
Dez 04 12:34:04 idefix kernel: RPC: Registered tcp-with-tls transport module.
Dez 04 12:34:04 idefix kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Dez 04 12:34:11 idefix systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Dez 04 12:34:31 idefix systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Dez 04 12:35:34 idefix systemd[1]: media-Bibliothek.mount: Mounting timed out. Terminating.
Dez 04 12:35:34 idefix systemd[1]: media-Bibliothek.mount: Mount process exited, code=killed, status=15/TERM
Dez 04 12:35:34 idefix systemd[1]: media-Bibliothek.mount: Failed with result 'timeout'.
Dez 04 12:35:34 idefix systemd[1]: media-Bibliothek.mount: Unit process 699 (mount.nfs) remains running after unit stopped.
Dez 04 12:35:34 idefix systemd[1]: Failed to mount /media/Bibliothek.
Dez 04 12:35:34 idefix systemd[1]: Dependency failed for Remote File Systems.
Dez 04 12:35:34 idefix systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
Dez 04 12:35:34 idefix systemd[1]: media-Musik.mount: Mounting timed out. Terminating.
Dez 04 12:35:34 idefix systemd[1]: media-Musik.mount: Mount process exited, code=killed, status=15/TERM
Dez 04 12:35:34 idefix systemd[1]: media-Musik.mount: Failed with result 'timeout'.
Dez 04 12:35:34 idefix systemd[1]: Failed to mount /media/Musik.
Dez 04 12:35:34 idefix systemd[1]: media-Videothek.mount: Mounting timed out. Terminating.
Dez 04 12:35:34 idefix systemd[1]: media-Videothek.mount: Mount process exited, code=killed, status=15/TERM
Dez 04 12:35:34 idefix systemd[1]: media-Videothek.mount: Failed with result 'timeout'.
Dez 04 12:35:34 idefix systemd[1]: Failed to mount /media/Videothek.
Dez 04 12:35:34 idefix systemd[1]: TPM2 PCR Barrier (User) was skipped because of an unmet condition check (ConditionPathExists=/sys/firmwa>
Dez 04 12:35:34 idefix systemd[1]: Starting Permit User Sessions...
Dez 04 12:35:34 idefix systemd[1]: Finished Permit User Sessions.
Dez 04 12:35:34 idefix systemd[1]: Reached target Multi-User System.
Dez 04 12:35:34 idefix systemd[1]: Starting GNOME Display Manager...
Dez 04 12:35:34 idefix systemd[1]: Started GNOME Display Manager.
Dez 04 12:35:34 idefix systemd[1]: Reached target Graphical Interface.
Dez 04 12:35:34 idefix systemd[1]: Startup finished in 2.618s (kernel) + 1min 35.708s (userspace) = 1min 38.326s.

____________________________________

Any hint is appreciated.

seth · 2023-12-04 12:00:58

1. code tags.
2. https://wiki.archlinux.org/title/Networ … ait-online

Arch Linux

#1 2023-04-02 11:52:07

NFS mounts started to fail at boot

#2 2023-04-02 12:40:49

Re: NFS mounts started to fail at boot

#3 2023-04-03 13:15:16

Re: NFS mounts started to fail at boot

#4 2023-04-03 13:30:35

Re: NFS mounts started to fail at boot

#5 2023-12-04 11:54:57

Re: NFS mounts started to fail at boot

#6 2023-12-04 12:00:58

Re: NFS mounts started to fail at boot

Board footer