[SOLVED] you need to reboot after nfs server install

SunBlade · 2024-08-13 19:59:14

Edit: failure can not be reproduced.
---

after a few hours of troubleshooting i finally found this thread: https://bbs.archlinux.org/viewtopic.php?id=152071

after you install the NFS Server you NEED to reboot. The server will refuse to work until you do.

i hope someone puts this message in the nfs-util package to save some headaches.

so, whenever systemd fails to start the nfs server and gives you some dependency errors, you probably forgot to restart.

Aug 13 21:10:23 systemd[1]: Dependency failed for NFSv4 ID-name mapping service.
Aug 13 21:10:23 systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start failed with result 'dependency'.
Aug 13 21:10:23 systemd[1]: Dependency failed for NFSv4 Client Tracking Daemon.
Aug 13 21:10:23 systemd[1]: nfsdcld.service: Job nfsdcld.service/start failed with result 'dependency'.
Aug 13 21:10:23 systemd[1]: Dependency failed for NFSv4 Mount Daemon.
Aug 13 21:10:23 systemd[1]: Dependency failed for NFSv4 server and services.
Aug 13 21:10:23 systemd[1]: nfsv4-server.service: Job nfsv4-server.service/start failed with result 'dependency'.
Aug 13 21:10:23 systemd[1]: nfsv4-exportd.service: Job nfsv4-exportd.service/start failed with result 'dependency'.
Aug 13 21:10:23 systemd[1]: Dependency failed for NFS server and services.
Aug 13 21:10:23 systemd[1]: Dependency failed for NFS Mount Daemon.
Aug 13 21:10:23 systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start failed with result 'dependency'.
Aug 13 21:10:23 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

Last edited by SunBlade (2024-10-15 20:39:12)

V1del · 2024-08-13 20:15:59

Unlikely, more likely that your install actually coincided with a kernel update on that same boot, leading to the kernel being unable to load modules for the nfs driver.

This is a common occurence, which can be mitigated with https://archlinux.org/packages/?q=kernel-modules-hook in oder to keep old modules around when a newer kernel gets installed.

Last edited by V1del (2024-08-13 20:16:32)

SunBlade · 2024-08-13 20:59:44

although i am very diligent in rebooting after kernel updates, i can not promise i missed a kernel update while installing all those packages over the past few days.
but i vaguely remember the last time i installed an ubuntu server i had the same problem.

i am no expert on this, i can only guess:
i suspect, since nfs has been integrated into the kernel, there is some flag, driver or whatnot that enables/disables nfs capabilities of the kernel at boot time.

hmmm... maybe i should treat nfs like a kernel driver in the future.

seth · 2024-08-13 23:05:42

nfs has been part of the kernel since 1993 and nothing "enables/disables nfs capabilities of the kernel at boot time" that would go away w/ a simple reboot.
V1del's suggestion is the by far most likely cause, if in the future you want to know for sure what's breaking some service,

systemctl status nfs-idmapd.service

and similar

SunBlade · 2024-10-15 20:51:49

seth wrote:

systemctl status nfs-idmapd.service

systemctl status and journalctl -au reported dependency errors for all nfs units. most of them did not elaborate any further about the dependency error. even the few ones which actually pointed towards the failing required unit, pointed at those dead end units.

i have tried to recreate this dependency failure with countless VMs, but try as i may nfs always worked like a charm.
even when i tried to intentionally break it by installing nfs after upgrading the kernel without rebooting, it just works. according to V1del, unless you use that pacman hook, this should not be possible?

since this is the second time i got that infuriatingly non-descriptive dependency error and both times from nfs, i believe this is no coincidence. but alas, i have not the foggiest clue on how to recreate that error.

seth · 2024-10-15 21:00:01

even when i tried to intentionally break it by installing nfs after upgrading the kernel without rebooting, it just works.

What exactly do you mean by "installing nfs"?
There're a bunch of userspace tools (notably for the server) that all have nothing to do w/ the kernel module and will work regardless of it.
What will not work is updating the kernel and then loading its nfs modules unless you've preserved the old modules for the running kernel.
So if you want to break anything, make sure the nfs modules aren't loaded (lsmod), update the kernel, make sure "modinfo nfs" get's you an error because the module could not be found and then try to utilize nfs.

SunBlade · 2024-10-15 22:57:43

seth wrote:

What exactly do you mean by "installing nfs"?

installing the nfs-utils package with pacman and enabling+starting the nfsv4-server.service with systemctl.

seth wrote:

There're a bunch of userspace tools (notably for the server) that all have nothing to do w/ the kernel module and will work regardless of it.

does that mean the kernel module is not part of the nfs-utils package? if it is not: does the kernel module get loaded by default?

seth wrote:

So if you want to break anything, make sure the nfs modules aren't loaded ...

i did not even think about testing the nfs kernel module directly. my tests revolved around the nfs-utils package, nfsv4-server.service and different /etc/exports configs. i tried on a replay of my original pacman.log and on different arch-installer presets (minimal, gui and server).

V1del · 2024-10-16 00:33:06

SunBlade wrote:

does that mean the kernel module is not part of the nfs-utils package? if it is not: does the kernel module get loaded by default?

The kernel module is part of the kernel, i.e. the linux package. The module gets loaded on demand, once you start some form of server application that needs the features of the module it will get loaded (check lsmod/dmesg). If it can't get loaded because you haven't loaded it yet and it's not present anymore (because of a kernel update) whatever operation you're trying there will fail.

Try going through seth's steps and ensuring the nfs module (or likely more correctly the nfsd module) is not loaded and can't be loaded (because of a kernel update -- or for testing purposes you could also blacklist it with a

install nfs /usr/bin/true

override or so https://wiki.archlinux.org/title/Kernel … acklisting

SunBlade · 2024-10-20 09:05:45

V1del wrote:

... (or likely more correctly the nfsd module) ...

yes! by breaking the nfsd module i get that infuriating dependency error.

now, lets play a game: which module did i actually break to get nfsd to fail?

starting the nfs server fails instantly.

#> systemctl start nfsv4-server.service
A dependency job for nfsv4-server.service failed. See 'journalctl -xe' for details.

systemctl and journalctl wont help beyond 'dependency'.

#> systemctl status nfsv4-server.service
○ nfsv4-server.service - NFSv4 server and services
     Loaded: loaded (/usr/lib/systemd/system/nfsv4-server.service; disabled; preset: disabled)
     Active: inactive (dead)

#> journalctl -b -au nfsv4-server.service
Oct 20 02:27:50 systemd[1]: Dependency failed for NFSv4 server and services.
Oct 20 02:27:50 systemd[1]: nfsv4-server.service: Job nfsv4-server.service/start failed with result 'dependency'.

and knowing which units exactly failed the chain does not help much either.

#> systemctl --failed
  UNIT                                 LOAD   ACTIVE SUB    DESCRIPTION
● proc-fs-nfsd.mount                   loaded failed failed NFSD configuration filesystem
● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

2 loaded units listed.

#> systemctl status proc-fs-nfsd.mount
× proc-fs-nfsd.mount - NFSD configuration filesystem
     Loaded: loaded (/usr/lib/systemd/system/proc-fs-nfsd.mount; static)
     Active: failed (Result: exit-code) since Sun 2024-10-20 02:27:50 CEST; 2min 53s ago
 Invocation: 691836153fd9489586e5ea17d8a59065
      Where: /proc/fs/nfsd
       What: nfsd
   Mem peak: 1.5M
        CPU: 3ms

#> journalctl -b -au proc-fs-nfsd.mount
Oct 20 02:27:50 systemd[1]: Mounting NFSD configuration filesystem...
Oct 20 02:27:50 mount[533]: mount: /proc/fs/nfsd: unknown filesystem type 'nfsd'.
Oct 20 02:27:50 mount[533]:        dmesg(1) may have more information after failed mount system call.
Oct 20 02:27:50 systemd[1]: proc-fs-nfsd.mount: Mount process exited, code=exited, status=32/n/a
Oct 20 02:27:50 systemd[1]: proc-fs-nfsd.mount: Failed with result 'exit-code'.
Oct 20 02:27:50 systemd[1]: Failed to mount NFSD configuration filesystem.

the kernel only tells us that the fs-nfsd module is properly loaded but not working as expected.

#> dmesg|tail
[    2.310406] kvm_amd: Virtual VMLOAD VMSAVE supported
[    2.310407] kvm_amd: Virtual GIF supported
[    2.998667] systemd-journald[331]: Received client request to flush runtime journal.
[    3.052587] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input6
[   48.468100] RPC: Registered named UNIX socket transport module.
[   48.468103] RPC: Registered udp transport module.
[   48.468104] RPC: Registered tcp transport module.
[   48.468105] RPC: Registered tcp-with-tls transport module.
[   48.468106] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   48.485721] request_module fs-nfsd succeeded, but still no fs?

looking at the entire journal around the error time, yields no additional information illuminating a culprit.

#> journalctl -a -S "1 minute ago"
Oct 20 02:27:50 systemd[1]: Listening on RPCbind Server Activation Socket.
Oct 20 02:27:50 systemd[1]: Reached target RPC Port Mapper.
Oct 20 02:27:50 systemd[1]: Mounting NFSD configuration filesystem...
Oct 20 02:27:50 systemd[1]: Mounting RPC Pipe File System...
Oct 20 02:27:50 systemd[1]: Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
Oct 20 02:27:50 systemd[1]: Starting GSSAPI Proxy Daemon...
Oct 20 02:27:50 systemd[1]: Starting Wait for Network to be Configured...
Oct 20 02:27:50 systemd[1]: Started GSSAPI Proxy Daemon.
Oct 20 02:27:50 systemd[1]: Mounted RPC Pipe File System.
Oct 20 02:27:50 systemd[1]: Reached target rpc_pipefs.target.
Oct 20 02:27:50 systemd[1]: RPC security service for NFS client and server was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
Oct 20 02:27:50 kernel: RPC: Registered named UNIX socket transport module.
Oct 20 02:27:50 kernel: RPC: Registered udp transport module.
Oct 20 02:27:50 kernel: RPC: Registered tcp transport module.
Oct 20 02:27:50 kernel: RPC: Registered tcp-with-tls transport module.
Oct 20 02:27:50 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Oct 20 02:27:50 mount[533]: mount: /proc/fs/nfsd: unknown filesystem type 'nfsd'.
Oct 20 02:27:50 mount[533]:        dmesg(1) may have more information after failed mount system call.
Oct 20 02:27:50 systemd[1]: proc-fs-nfsd.mount: Mount process exited, code=exited, status=32/n/a
Oct 20 02:27:50 systemd[1]: proc-fs-nfsd.mount: Failed with result 'exit-code'.
Oct 20 02:27:50 systemd[1]: Failed to mount NFSD configuration filesystem.
Oct 20 02:27:50 systemd[1]: Dependency failed for NFSv4 Mount Daemon.
Oct 20 02:27:50 systemd[1]: Dependency failed for NFSv4 server and services.
Oct 20 02:27:50 systemd[1]: nfsv4-server.service: Job nfsv4-server.service/start failed with result 'dependency'.
Oct 20 02:27:50 systemd[1]: nfsv4-exportd.service: Job nfsv4-exportd.service/start failed with result 'dependency'.
Oct 20 02:27:50 systemd[1]: Dependency failed for NFS Mount Daemon.
Oct 20 02:27:50 systemd[1]: Dependency failed for NFS server and services.
Oct 20 02:27:50 systemd[1]: Dependency failed for NFSv4 ID-name mapping service.
Oct 20 02:27:50 systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start failed with result 'dependency'.
Oct 20 02:27:50 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Oct 20 02:27:50 systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start failed with result 'dependency'.
Oct 20 02:27:50 systemd[1]: Dependency failed for NFSv4 Client Tracking Daemon.
Oct 20 02:27:50 systemd[1]: nfsdcld.service: Job nfsdcld.service/start failed with result 'dependency'.
Oct 20 02:27:50 kernel: request_module fs-nfsd succeeded, but still no fs?

so, nothing obvious points out that a kernel module is broken and that it is nfsd itself.

#> cat /etc/modprobe.d/blacklist.conf
install nfsd /usr/bin/true

as long as i know that the nfsd module is broken, i can see the subtle clues in the error logs. but without that knowledge, i would not be able to connect the dots.
and when i tried breaking other modules, i also got only subtle clues in the error logs. even when a module file physically does not exist, the kernel gives no error message.
does the kernel log failed module loads somewhere else? or not at all? or do i need to set some debug var? well, i probably need to read up on that.

maybe next time i run into dependency errors, i will look out for those subtle clues in order to deduce if & which module fails.
and after seeing all those subtle clues by breaking that much modules, i am now convinced that you are right and i forgot to reboot between kernel update and nfs server install.

seth · 2024-10-20 11:32:38

Oct 20 02:27:50 mount[533]: mount: /proc/fs/nfsd: unknown filesystem type 'nfsd'

Unknown filesystem type is a strong hint.

the kernel only tells us that the fs-nfsd module is properly loaded but not working as expected.

Does it?
https://patchwork.kernel.org/project/li … ernel.org/
The proper check would be

lsmod | grep nfs

SunBlade · 2024-10-21 22:15:23

i still can't wrap my head around this behaviour.

i expect neon blinking error messages, if a file, which is crucial for further operations and was clearly requested, is missing or malformed.
yet the kernel stays silent, if a module ...

does not exist
is not a valid executable binary
does not provide operation critical functionality

the only error message i managed to get is: when a module failed to load, because it tried to dynamically load functions from another module, which did not exist. but i am not sure if the kernel logged those errors or the module itself.

after doing this:

#> cat /etc/modprobe.d/test.conf
install auth_rpcgss /dev_random_trash

i would like the module auth_rpcgss to be at least mentioned once in here:

#> journalctl -a -S "3 second ago"
Oct 21 15:16:55 systemd[1]: Listening on RPCbind Server Activation Socket.
Oct 21 15:16:55 systemd[1]: Reached target RPC Port Mapper.
Oct 21 15:16:55 systemd[1]: Mounting NFSD configuration filesystem...
Oct 21 15:16:55 systemd[1]: Mounting RPC Pipe File System...
Oct 21 15:16:55 systemd[1]: Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
Oct 21 15:16:55 systemd[1]: Starting GSSAPI Proxy Daemon...
Oct 21 15:16:55 systemd[1]: Starting Wait for Network to be Configured...
Oct 21 15:16:55 systemd[1]: Started GSSAPI Proxy Daemon.
Oct 21 15:16:55 systemd[1]: Mounted RPC Pipe File System.
Oct 21 15:16:55 kernel: RPC: Registered named UNIX socket transport module.
Oct 21 15:16:55 kernel: RPC: Registered udp transport module.
Oct 21 15:16:55 kernel: RPC: Registered tcp transport module.
Oct 21 15:16:55 kernel: RPC: Registered tcp-with-tls transport module.
Oct 21 15:16:55 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Oct 21 15:16:55 systemd[1]: Reached target rpc_pipefs.target.
Oct 21 15:16:55 systemd[1]: RPC security service for NFS client and server was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
Oct 21 15:16:55 mount[537]: mount: /proc/fs/nfsd: unknown filesystem type 'nfsd'.
Oct 21 15:16:55 mount[537]:        dmesg(1) may have more information after failed mount system call.
Oct 21 15:16:55 systemd[1]: proc-fs-nfsd.mount: Mount process exited, code=exited, status=32/n/a
Oct 21 15:16:55 systemd[1]: proc-fs-nfsd.mount: Failed with result 'exit-code'.
Oct 21 15:16:55 systemd[1]: Failed to mount NFSD configuration filesystem.
Oct 21 15:16:55 systemd[1]: Dependency failed for NFS server and services.
Oct 21 15:16:55 systemd[1]: Dependency failed for NFSv4 ID-name mapping service.
Oct 21 15:16:55 systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start failed with result 'dependency'.
Oct 21 15:16:55 systemd[1]: Dependency failed for NFS Mount Daemon.
Oct 21 15:16:55 systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start failed with result 'dependency'.
Oct 21 15:16:55 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Oct 21 15:16:55 systemd[1]: Dependency failed for NFSv4 server and services.
Oct 21 15:16:55 systemd[1]: Dependency failed for NFSv4 Mount Daemon.
Oct 21 15:16:55 systemd[1]: nfsv4-exportd.service: Job nfsv4-exportd.service/start failed with result 'dependency'.
Oct 21 15:16:55 systemd[1]: nfsv4-server.service: Job nfsv4-server.service/start failed with result 'dependency'.
Oct 21 15:16:55 systemd[1]: Dependency failed for NFSv4 Client Tracking Daemon.
Oct 21 15:16:55 systemd[1]: nfsdcld.service: Job nfsdcld.service/start failed with result 'dependency'.

nothing in here screams "kernel module failure" at me.

seth wrote:

Unknown filesystem type is a strong hint.

to you this may look like a neon sign on an empty country road. but to a newbie like me, this only looks like one of the fancier storefronts in a bustling mall.
without the knowledge that nfsd is also a kernel module and /proc being a bridge between the kernel and userspace, this would look even more like noise than a critical error message.
i hope someday i am able to see what you see, but problems like this remind me that i still have a long way in front of me.

to come to the conclusion that i need to check kernel modules, i first need to be aware that ...

the kernel does not complain about module errors (this was shocking news for me until a few days ago)
my command just now, needs a specific kernel module (fortunately a few years ago while configuring samba i accidentally learned that nfs uses a kernel module)

if i know those things, i can start checking kernel modules. but not with lsmod. lsmod shows only successfully loaded modules, not failed ones.
first i need to find out which module my command tried to load. and after i guessed or googled correctly, modprobe elaborates on what exactly went wrong.

#> modprobe -v nfsd
install /dev_random_trash
sh: line 1: /dev_random_trash: cannot execute binary file: Exec format error
modprobe: ERROR: Error running install command '/dev_random_trash' for module auth_rpcgss: retcode 126
modprobe: ERROR: could not insert 'nfsd': Invalid argument

i whish the kernel would print those errors.

seth wrote:

Does it?

bad wording. what i meant was that the module was a proper executable binary, but was not working as expected.
modules are considered loaded when the kernel puts them in /proc/modules.

seth · 2024-10-22 11:39:17

If you try to load a module that's not available, modprobe will complain about that:

modprobe gnarf                    
modprobe: FATAL: Module gnarf not found in directory /lib/modules/…

The "problem" here is that the module in question is a filesystem driver what will get you

mount -t gnarf /dev/null /mnt
mount: /mnt: unknown filesystem type 'gnarf'.
       dmesg(1) may have more information after failed mount system call.

when mount tries to use, but cannot find it.
The filesystem driver could also be a fuse one which is why mount complaining about a missing kernel module when it cannot find the mount type would be misleading.

Arch Linux

#1 2024-08-13 19:59:14

[SOLVED] you need to reboot after nfs server install

#2 2024-08-13 20:15:59

Re: [SOLVED] you need to reboot after nfs server install

#3 2024-08-13 20:59:44

Re: [SOLVED] you need to reboot after nfs server install

#4 2024-08-13 23:05:42

Re: [SOLVED] you need to reboot after nfs server install

#5 2024-10-15 20:51:49

Re: [SOLVED] you need to reboot after nfs server install

#6 2024-10-15 21:00:01

Re: [SOLVED] you need to reboot after nfs server install

#7 2024-10-15 22:57:43

Re: [SOLVED] you need to reboot after nfs server install

#8 2024-10-16 00:33:06

Re: [SOLVED] you need to reboot after nfs server install

#9 2024-10-20 09:05:45

Re: [SOLVED] you need to reboot after nfs server install

#10 2024-10-20 11:32:38

Re: [SOLVED] you need to reboot after nfs server install

#11 2024-10-21 22:15:23

Re: [SOLVED] you need to reboot after nfs server install

#12 2024-10-22 11:39:17

Re: [SOLVED] you need to reboot after nfs server install

Board footer