You are not logged in.

#1 2021-01-09 20:15:34

marekhwd
Member
Registered: 2018-02-22
Posts: 7

NFS, shutdown/reboot, A stop job is running for /mnt/foo

Previously https://bbs.archlinux.org/viewtopic.php?id=234084.

I have the following NFS mount in /etc/fstab:

nas.mynet.local:/mnt/storage /mnt/storage nfs4 _netdev,noauto,noatime,sec=krb5p 0 0

I mount this filesystem manually when I need it. I don't use x-systemd.automount because if I'm on different network listing /mnt hangs, probably because automounter kicks when /mnt/storage is stat()ed. Adding this option also doesn't help the issue I'm having in any observable way.

The issue is: Every time I shutdown or reboot the system with a NFS (or CIFS) filesystem mounted, systemd waits 90 seconds for something with message:

A stop job is running for /mnt/storage (1s / 1min 30s)

What is the problem? How can I fix it?

Note that I have sec=krb5p in mount options. I'm using Kerberos for authentication, and (authenticated) encryption of NFS traffic. The old-style authentication via UIDs can be enabled by sec=sys, which also doesn't have any effect on the issue. What may be different from other reported issues on Internet is that I'm using NFSv4.

This is how my mount point looks on running system:

nas.mynet.local:/mnt/storage on /mnt/storage type nfs4 (rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.1.10,local_lock=none,addr=192.168.1.1,_netdev)

Systemd unit metadata (dependencies, ordering, ...) obtained from systemctl show mnt-storage.mount: https://pastebin.com/raw/pr2fsJ7w

I use NetworkManager and NetworkManager-wait-online.service is enabled and active after boot.

Adding systemd.requires=network-online.target to mount options has no effect.

I have quite strict firewall rules for incoming traffic set up by nftables.service, but I don't see how that would be affecting this problem.

It happens with both NFS and CIFS, so I assume it's problem with ordering of service shutdowns. I don't seem to have the ability to scroll in systemd boot log but hopefully I see everything it prints from shutdown to the wait for unmount, and I see no network unit is stopped there. After the timeout expires and systemd gives up on the mount, bunch of other units are stopped but it happens too fast before reboot to read anything.

It's really frustrating. I learned to manually unmount network filesystem before shutdown but sometimes I forget, and generally it's inconvenient. It seems I'm not the only one having this problem but there's not much useful information to track this problem down. Thanks for all your suggestions.

Offline

#2 2021-01-12 20:02:02

EdeWolf
Member
Registered: 2016-01-06
Posts: 79

Re: NFS, shutdown/reboot, A stop job is running for /mnt/foo

I'd rather suspect, some process may still access the mount, than an ordering problem, but I am no expert, so I may be completely off track

But first I would try to add following to /etc/systemd/logind.conf:

# User settings:
KillUserProcesses=yes
KillExcludeUsers=root
RemoveIPC=yes

Be aware, this also may kill tmux or screen sessions after logout.

You may also create an unit file, that does an "lsof | grep storage > somelog" during shutdown, to see, wether not some process will use this path.

Not an ideal solution, but what if you create a mnt-storage.mount unit instead of an fstab entry? This way you could easily control the ordering and see, wether this is the reason.

Or, just create an unit file, that will come up with ExecStart=/bin/true (RemainAfterExit=yes) and with an ExecStop=/usr/bin/umount /mnt/storage and order it the way you want. Just an idea to play around.

No real help, I know, but perhaps hints for troubleshooting.

Offline

#3 2021-02-08 02:23:04

marekhwd
Member
Registered: 2018-02-22
Posts: 7

Re: NFS, shutdown/reboot, A stop job is running for /mnt/foo

Thanks for the reply EdeWolf. I've finally found some time to take another look. Problem turned out to be something completely different.

As I said I use NetworkManager and I'm always on one of few Wi-Fi connections. By closely examining shutdown sequence in journal log I noticed that NetworkManager tears down active Wi-Fi connection long before stopping remote-fs.target, so when it's time to unmount /mnt/storage there is no network. However, it is not the case that shutdown ordering of services is wrong as in this case. In my case shutdown ordering is proper:

% journalctl -b-1 | egrep 'Stop.*(D-Bus|Network M|Remote F)'
Feb 05 13:21:36 localhost systemd[560]: Stopping D-Bus User Message Bus...
Feb 05 13:21:36 localhost systemd[560]: Stopped D-Bus User Message Bus.
Feb 05 13:21:37 localhost systemd[1]: Stopped target Remote File Systems.
Feb 05 13:23:07 localhost systemd[1]: Stopped target Remote File Systems (Pre).
Feb 05 13:23:07 localhost systemd[1]: Stopped Network Manager Wait Online.
Feb 05 13:23:07 localhost systemd[1]: Stopping Network Manager...
Feb 05 13:23:07 localhost systemd[1]: Stopped Network Manager.
Feb 05 13:23:07 localhost systemd[1]: Stopping D-Bus System Message Bus...
Feb 05 13:23:07 localhost systemd[1]: Stopped D-Bus System Message Bus.

I set up all my Wi-Fi connections via nm-applet and nm-connection-editor (i.e. by clicking). I've noticed that all of them have All users may connect to this network checkbox unchecked. Not sure whether it's the default or I did it. It turns out that when this option is checked, active Wi-Fi connection stays active while unmounting. So that's it, single checkbox change. If it's unchecked, then during shutdown something decides that since I'm being logged out it's time to also tear down my Wi-Fi connection, probably by instructing NetworkManager via D-Bus.

To my dismay I discovered that this makes connection details including password available to any user account running on the box (sudo -u nobody nmcli device wifi show-password). One solution is to go Wi-Fi Security tab, clicking the person icon in the Password field, and selecting Store the password only for this user (credits to ArchWiki). I don't care about Wi-Fi connecting only after login so this is fine for me. Another, probably better solution would be to restrict access to NetworkManager D-Bus API via PolicyKit.

Last edited by marekhwd (2021-02-08 02:28:44)

Offline

#4 2022-03-07 07:52:37

seands
Member
Registered: 2020-09-06
Posts: 14

Re: NFS, shutdown/reboot, A stop job is running for /mnt/foo

@marekhwd, thanks, checking the WiFi checkbox worked for me too! You might consider to add this solution to the Arch wiki?

And I guess mark this as solved?

Last edited by seands (2022-03-07 07:53:11)

Offline

Board footer

Powered by FluxBB