You are not logged in.

#1 2017-07-31 17:27:04

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

[SOLVED] NFS mounts fail on startup (netctl needed wait-online)

I have a couple of NFS mounts in /etc/fstab :

koa.int.boulder.nso.edu:/export/data/isoon /isoon nfs auto,nouser,noexec,ro 0 0
isilon-sc.nispdc.nso.edu:/ifs/nisp   /isilon_ro nfs auto,nouser,noexec,ro 0 0

These have been working for some time, however after updating Arch today with "pacman -Syu" the boot message tells me that they have failed and that I should consult "systemctl status". When I do that I see this :

# systemctl status isilon_ro.mount
   isilon_ro.mount - /isilon_ro
   Loaded: loaded (/etc/fstab; generated; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-07-31 11:10:16 MDT; 3min 23s ago
    Where: /isilon_ro
     What: isilon-sc.nispdc.nso.edu:/ifs/nisp
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 364 ExecMount=/usr/bin/mount isilon-sc.nispdc.nso.edu:/ifs/nisp /isilon_ro -t nfs -o nouser,noexec,ro (code=exited, status=32)

Jul 31 11:10:14 spsc-nso19-12 systemd[1]: Mounting /isilon_ro...
Jul 31 11:10:16 spsc-nso19-12 systemd[1]: isilon_ro.mount: Mount process exited, code=exited status=32
Jul 31 11:10:16 spsc-nso19-12 systemd[1]: Failed to mount /isilon_ro.
Jul 31 11:10:16 spsc-nso19-12 systemd[1]: isilon_ro.mount: Unit entered failed state.

Oddly, if I then try to mount them with "mount -a" or with the command from the above "systemctl status" :

/usr/bin/mount isilon-sc.nispdc.nso.edu:/ifs/nisp /isilon_ro -t nfs -o nouser,noexec,ro

Then that works with no error message and the mount is there :

# /usr/bin/mount isilon-sc.nispdc.nso.edu:/ifs/nisp /isilon_ro -t nfs -o nouser,noexec,ro
# systemctl status isilon_ro.mount
  isilon_ro.mount - /isilon_ro
   Loaded: loaded (/etc/fstab; generated; vendor preset: disabled)
   Active: active (mounted) (Result: exit-code) since Mon 2017-07-31 11:22:20 MDT; 4s ago
    Where: /isilon_ro
     What: isilon-sc.nispdc.nso.edu:/ifs/nisp
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 364 ExecMount=/usr/bin/mount isilon-sc.nispdc.nso.edu:/ifs/nisp /isilon_ro -t nfs -o nouser,noexec,ro (code=exited, status=32)

I'm a little at a loss given that it was working before, and without an error message to pursue on manually mounting. Any thoughts appreciated! Thanks!

Last edited by nilesOien (2017-08-02 15:38:54)


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#2 2017-07-31 21:00:03

seth
Member
Registered: 2012-09-03
Posts: 49,951

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Try adding "x-systemd.requires=network.target" to the mount options.

Edit: See https://wiki.archlinux.org/index.php/NF … th_systemd - notably if you're using networkmanager.

Last edited by seth (2017-07-31 21:01:16)

Offline

#3 2017-07-31 21:40:02

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Thanks, unfortunately no change. To be clear I changed /etc/fstab from :

isilon-sc.nispdc.nso.edu:/ifs/nisp   /isilon_ro nfs auto,nouser,noexec,ro 0 0
koa.int.boulder.nso.edu:/export/data/isoon /isoon nfs auto,nouser,noexec,ro 0 0

To :

isilon-sc.nispdc.nso.edu:/ifs/nisp /isilon_ro  nfs x-systemd.requires=network.target,auto,nouser,noexec,ro 0 0
koa.int.boulder.nso.edu:/export/data/isoon /isoon nfs x-systemd.requires=network.target,auto,nouser,noexec,ro 0 0

Things unfortunately remained the same, the disks are not mounted at startup but subsequently "mount -a" picks them right up. I do also see this on startup :

Dependency failed for network manager wait onlie

I may have that slightly wrong, I'm typing it from notes that I made as it passed by on boot. I think it's been doing that for a while, with no ill effects that I can see, but maybe that is significant here?


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#4 2017-07-31 21:43:23

seth
Member
Registered: 2012-09-03
Posts: 49,951

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Did you enable the NetworkManager-wait-online.service? What's its status?

Offline

#5 2017-07-31 22:08:19

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

I did enable that, but it seems like it's not starting :

# systemctl enable NetworkManager-wait-online.service
Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service → /usr/lib/systemd/system/NetworkManager-wait-online.service.

# systemctl status NetworkManager-wait-online.service
  NetworkManager-wait-online.service - Network Manager Wait Online
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager-wait-online.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:nm-online(1)

Jul 31 15:50:12 spsc-nso19-12 systemd[1]: Dependency failed for Network Manager Wait Online.
Jul 31 15:50:12 spsc-nso19-12 systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start failed 

I (possibly stupidly) tried "systemctl start NetworkManager-wait-online.service", which told me "A dependency job for NetworkManager-wait-online.service failed. See 'journalctl -xe' for details".

journalctl -xe gave me :

-- Subject: Unit NetworkManager-wait-online.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit NetworkManager-wait-online.service has failed.
-- 
-- The result is dependency.
Jul 31 15:55:17 spsc-nso19-12 systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start failed with result 'dependency'.
Jul 31 15:55:21 spsc-nso19-12 usbmuxd[279]: [15:55:21.769][1] config_get_device_record: failed to read '/var/lib/lockdown/db964cb3acf0ebc61ea4e10fa5fcbc58a8c8a428.plist': No such file or directory
Jul 31 15:55:26 spsc-nso19-12 usbmuxd[279]: [15:55:26.769][1] config_get_device_record: failed to read '/var/lib/lockdown/db964cb3acf0ebc61ea4e10fa5fcbc58a8c8a428.plist': No such file or directory

I've no idea if the "config_get_device_record" messages are relevant, I kind of suspect not, they seem to be coming pretty regularly. I hope I'm making sense - this isn't an area that I deal with very regularly I'm afraid.

Thanks again!


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#6 2017-07-31 22:28:48

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Well, after all that, it turned out NetworkManager wasn't running. The substance of the solution was :

systemctl start NetworkManager.service
systemctl enable NetworkManager.service
systemctl start NetworkManager-wait-online.service
systemctl enable NetworkManager-wait-online.service

The mounts seem fine, I'm marking as [SOLVED] -

Thanks, I needed this discussion to point me in the right direction!


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#7 2017-08-01 05:42:50

seth
Member
Registered: 2012-09-03
Posts: 49,951

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Well, after all that, it turned out NetworkManager wasn't running.

How did you maintain the network connection before?
Make sure you're not running concurrent network managing services (ie. NM along dhcpcd.service, netctl, wicd, etcetc.) - you'll need to handle this differently then (bu know that this happens because the NFS mount was attempted pre-network, thus failed)

Offline

#8 2017-08-01 17:06:42

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Good point, and it never occurred to me. I rarely set up machines, and forgot the potential for conflict between network controllers.

"systemctl --type=service" did indeed show that I had both NetworkManager.service and netctl@nso_network.service running with NetworkManager-wait-online.service active and exited. I decided that wanted to move to NetworkManager, so I stopped and disabled netctl@nso_network.service and then went through and set up NetworkManager through the gnome GUI as root (I think I ran into a known NM bug around setting up a static IP). It took a few reboots, and I did find that I had to fall back to dhcpd for a bit just to be able to read online documentation, but now I'm running only NetworkManager.

I should have paid more attention to the fact that the network icon at the top right of gnome was a question mark! With no network controller conflicts, it now appears as a wired connection icon, and the NFS mounts seem to be coming up correctly.

Thanks again,

Niles.


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#9 2017-08-01 22:58:19

BlackMastermind
Member
Registered: 2017-01-17
Posts: 45

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

I'm having the same issue here: NFS shares do not get mounted at boot anymore since I rebooted this morning, and some network services are failing to start because they cannot bind to the specific IP that they expect to be present. So it seems that some network dependent services are launched before the network is actually online. I also noticed that during the boot sequence, there is no short pause anymore when the network is being brought online via dhcp.

After some troubleshooting, I found that it is related to the netctl 1.12 -> 1.13 update of a few days ago. In that update, the systemd unit file /usr/lib/systemd/system/netctl@.service has been changed, most notably Type=oneshot has been changed to Type=notify.

Now I'm not a systemd unit file expert, so I don't know the details on what notify is supposed to do as opposed to oneshot, and why it causes the issue I'm seeing, but if I add the following lines to the unit file that configures my network (/etc/systemd/system/netctl@ethernet\\x2ddhcp.service), everything works normally again:

[Service]
Type=oneshot

Offline

#10 2017-08-02 00:01:06

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Hmmm. Based on that, I'm removing the [SOLVED] tag and changing the subject line a bit to reflect the fact that it seems like I didn't really solve it, I worked around it.


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#11 2017-08-02 06:45:05

seth
Member
Registered: 2012-09-03
Posts: 49,951

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

"Surprise"
https://git.archlinux.org/netctl.git/co … fa450d58d4

systemctl enable netctl-wait-online

Should do. The behavior seems now similar to networkmanager (itr. ;-) and that should probably have been documented in the wiki (at least)

Offline

#12 2017-08-02 15:34:39

nilesOien
Member
From: Boulder, CO, USA
Registered: 2015-05-08
Posts: 77

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

Nice work. I think it's worth announcing this. Putting out an alert of some form may stop a lot of reported issues, or at least nip them in the bud.

I'm not sure how best to tell people that they now may need netctl-wait-online. What is a good way to do that?

EDIT : I'm changing the subject tag back to [SOLVED] since this seems fairly definitive.

Last edited by nilesOien (2017-08-02 15:37:43)


-- "Make it as simple as possible, but no simpler" - Albert Einstein

Offline

#13 2017-08-02 21:07:08

BlackMastermind
Member
Registered: 2017-01-17
Posts: 45

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

The NFS issue was solved for me by:

systemctl enable netctl-wait-online

However, I also have an ssh daemon on this host that is configured to listen only on a specific interface, via the ListenAddress parameter. Reason being is that this is a multihomed system, and I don't want to accidentally expose sshd on other interfaces.

Anyway, this sshd still failed to start because in the /usr/lib/systemd/system/sshd.service unit file it is set to run after network.target, while it should actually run after network-online.target. The result is: it runs too early and fails, because the IP address it is expecting doesn't exist yet.

I worked around this by creating a unit file /etc/systemd/system/sshd.service with the following content, which overrides this particular setting in the original unit file:

.include /usr/lib/systemd/system/sshd.service

[Unit]
After=network-online.target

Last edited by BlackMastermind (2017-08-02 21:16:42)

Offline

#14 2017-08-03 02:10:50

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

Re: [SOLVED] NFS mounts fail on startup (netctl needed wait-online)

For those using systemd-networkd (as opposed to netctl or NetworkManager) to manage network connections, here is how I solved this issue.

systemd-networkd-wait-online.service is the systemd-networkd "wait-online" equivalent and is supposed to be enabled automatically when systemd-networkd is enabled/installed.  For some reason it was disabled on my system -- perhaps because systemd-networkd-wait-online.service was introduced after I installed systemd-networkd?

Regardless, enabling this didn't completely solve the problem for me as systemd-networkd-wait-online.service would time out on startup (after 120 seconds).  However, the NFS drive would successfully mount after this service timed out. 

Here is the output of journalctl --boot=0 | grep systemd-networkd that shows this happening:

Aug 02 16:55:50 archlinux systemd-networkd[225]: Enumeration completed
Aug 02 16:55:51 archlinux systemd-networkd-wait-online[227]: ignoring: lo
Aug 02 16:55:51 archlinux systemd-networkd-wait-online[227]: ignoring: lo
Aug 02 16:55:51 archlinux systemd-networkd-wait-online[227]: ignoring: lo
Aug 02 16:55:51 archlinux systemd-networkd-wait-online[227]: ignoring: lo
Aug 02 16:55:53 archlinux systemd-networkd[225]: eth0: Renamed to eno1
Aug 02 16:55:53 archlinux systemd-networkd[225]: eno1: IPv6 successfully enabled
Aug 02 16:55:54 archlinux systemd-networkd[225]: wlan0: Renamed to wlo1
Aug 02 16:55:54 archlinux systemd-networkd[225]: wlo1: IPv6 successfully enabled
Aug 02 16:55:58 archlinux systemd-networkd[225]: wlo1: Gained carrier
Aug 02 16:55:59 archlinux systemd-networkd[225]: wlo1: Gained IPv6LL
Aug 02 16:56:01 archlinux systemd-networkd[225]: wlo1: DHCPv4 address 192.168.14.230/24 via 192.168.14.1
Aug 02 16:56:10 archlinux systemd-networkd[225]: wlo1: Configured
Aug 02 16:56:10 archlinux systemd-networkd-wait-online[227]: ignoring: lo
Aug 02 16:57:51 archlinux systemd-networkd-wait-online[227]: Event loop failed: Connection timed out
Aug 02 16:57:51 archlinux systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 16:57:51 archlinux systemd[1]: systemd-networkd-wait-online.service: Unit entered failed state.
Aug 02 16:57:51 archlinux systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.

After some reading of man systemd-networkd-wait-online it seems as though this service is waiting for all interfaces (eg. wired & wireless) to be configured.  So even though my wireless interface had an IP, the wait-online service was waiting on my wired interface as well -- which rarely gets plugged in on my laptop so doesn't get an IP.

You can tell systemd-networkd-wait-online.service to only wait on a specific interface and ignore others.

systemctl edit systemd-networkd-wait-online

This will create a new file called /etc/systemd/system/systemd-networkd-wait-online.service.d/override.conf.  I set the contents of this file as follows:

#ignore all interfaces except wlo1
#
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=wlo1

I rebooted my system to test it and the NFS drive was mounted right away.  Running journalctl --boot=0 | grep networkd after startup again shows the wired interface (eno1) being ignored now.

Aug 02 18:48:22 archlinux systemd-networkd[222]: Enumeration completed
Aug 02 18:48:24 archlinux systemd-networkd[222]: eth0: Renamed to eno1
Aug 02 18:48:24 archlinux systemd-networkd[222]: eno1: IPv6 successfully enabled
Aug 02 18:48:24 archlinux systemd-networkd[222]: wlan0: Renamed to wlo1
Aug 02 18:48:24 archlinux systemd-networkd[222]: wlo1: IPv6 successfully enabled
Aug 02 18:48:30 archlinux systemd-networkd[222]: wlo1: Gained carrier
Aug 02 18:48:31 archlinux systemd-networkd[222]: wlo1: Gained IPv6LL
Aug 02 18:48:32 archlinux systemd-networkd[222]: wlo1: DHCPv4 address 192.168.14.230/24 via 192.168.14.1
Aug 02 18:48:42 archlinux systemd-networkd[222]: wlo1: Configured
Aug 02 18:48:42 archlinux systemd-networkd-wait-online[228]: ignoring: lo
Aug 02 18:48:42 archlinux systemd-networkd-wait-online[228]: ignoring: eno1

This is a suitable solution for me as I'm using the wireless conection 99% of the time.  If you can't afford to ignore the wired interface as I did, you can override the time out to be shorter perhaps.  See man systemd-networkd-wait-online for the option to do this.

HTH.

Offline

Board footer

Powered by FluxBB