NFS & Kerberos not working: export does not exist

pvdl1984 · 2021-12-27 16:39:27

I'm trying to get Kerberos working for NFS. But I get very unclear errors.

My exports file:

/nas             *(rw,sync,no_subtree_check,sec=krb5p,fsid=root)
/nas/personal    *(rw,no_subtree_check,sec=krb5p,sync)

If I set sec=sys, everything works fine:

sudo mount -vv sup.home:/ /mnt                                                                                                                                                                             
mount.nfs: timeout set for Mon Dec 27 17:30:55 2021
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.2.201,clientaddr=192.168.2.16'

If I set sec=krb5p, it is not working:

mount.nfs: timeout set for Mon Dec 27 17:31:52 2021
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.2.201,clientaddr=192.168.2.16'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=192.168.2.201'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.2.201 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.2.201 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Permission denied
mount.nfs: Operation not permitted

According to the server logs, all of a sudden it is not exported:

Dec 27 16:29:52 sup rpc.mountd[831]: refused mount request from 192.168.2.16 for / (/): not exported
Dec 27 16:29:56 sup rpc.mountd[831]: authenticated mount request from 192.168.2.16:735 for /nas (/nas)

The second line is when I try to mount /nas instead , because the logs claim the share does not exist. That get's the same error on the client.

I have also tried to look at other causes, the main thing which is not clear to me is if hostnames are working correctly (I don't see that mentioned anywhere in the logs, I have added hostnames to /etc/hosts as unfortunately the internal dns server is disabled).

Last edited by pvdl1984 (2021-12-27 16:49:52)

seth · 2021-12-27 21:16:16

Server and client seem to be off by an hour?
krb5 is time sensitive, make sure the clocks are aligned.

pvdl1984 · 2021-12-28 10:30:51

The timezone is different (server on utc, laptop on local timezone), the time is the same though. This would also give a different error if I'm correct (something about time skew) and would also mean that using the command line utility kinit wouldn't work, but this will get a ticket just fine.

seth · 2021-12-28 13:35:20

would also mean that using the command line utility kinit wouldn't work

Did you try the other krb5 protocols (notably pure krb5)?

pvdl1984 · 2021-12-28 15:33:08

seth wrote:

would also mean that using the command line utility kinit wouldn't work
Did you try the other krb5 protocols (notably pure krb5)?

I didn't try that yet. Now I both tried krb5 and krb5i, both are not working.

I also just verified the time:

# client
$ timedatectl status                                                                                                                                                                                            
               Local time: Tue 2021-12-28 16:31:27 CET
           Universal time: Tue 2021-12-28 15:31:27 UTC
                 RTC time: Tue 2021-12-28 15:31:27
                Time zone: Europe/Amsterdam (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
# server
$  timedatectl status
               Local time: Tue 2021-12-28 15:31:25 UTC
           Universal time: Tue 2021-12-28 15:31:25 UTC
                 RTC time: Tue 2021-12-28 15:31:26
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

The 2 second time difference is because I manually ran the commands.

seth · 2021-12-28 15:50:18

Can you

nfs-ls nfs://sup.home/nas

?
What do the mount options look like? Does adding/enforcing "mountproto=tcp" or "mountproto=udp" help?

For more obscure stuff, do you maybe use jumbo frames on either end or a VPN (though the IPs don't suggest the latter, the TZ makes me wonder)

pvdl1984 · 2021-12-28 16:21:02

seth wrote:

Can you
nfs-ls nfs://sup.home/nas
?
What do the mount options look like? Does adding/enforcing "mountproto=tcp" or "mountproto=udp" help?
For more obscure stuff, do you maybe use jumbo frames on either end or a VPN (though the IPs don't suggest the latter, the TZ makes me wonder)

$ sudo nfs-ls nfs://sup.home/nas                                                                                                                                                                               
Failed to opendir("") opendir call failed with "NFS: READDIRPLUS of  failed with NFS3ERR_ACCES(-13)"

$ sudo nfs-ls nfs://sup.home/                                                                                                                                                                                 
Failed to mount nfs share : mount_cb: RPC error: Mount failed with error MNT3ERR_ACCES(13) Permission denied(13)

Seems to fail as well. This is all on a fairly basic network setup, no VPN, no jumbo frames. The TZ difference is purely a choice by me.

Mounting with `mountproto=tcp` or `mountproto=udp` Is either giving "access denied by server while mounting sup.home:/personal" or if I prefix with "/nas" I get "an incorrect mount option was specified". If I add :sec=krb5" I will get "Invalid argument".

The main thing I'm wondering is why without Kerberos I need to request "/personal" and with Kerberos I need to request "/nas/personal" as it will complain about "not exported" in the server log. One other thing I noticed is that if I capture with wireshark, I see the client sending the short hostname instead of the FQDN (although I might be missing something).

Last edited by pvdl1984 (2021-12-28 16:30:21)

pvdl1984 · 2021-12-28 16:46:07

Without having read the full nfs4 spec. My guess is that the client doesn't know Kerberos (it seems to connect, ask security info, then closes). Again, might be missing some details as I have never read the nfs 4 protocol spec. But I also "Invalid argument" when specifying

-osec=krb5

.

pvdl1984 · 2021-12-28 16:59:58

I managed to miss one important detail (enable `nfs-client.target` on the client). Which doesn't solve the issue, but at least gives me more information.

Now the server gives me the following:

Dec 28 16:55:11 sup krb5kdc[660]: TGS_REQ (3 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), DEPRECATED:des3-cbc-sha1(16)}) 192.168.2.16: UNKNOWN_SERVER: authtime 0, etypes {rep=UNSUPPORTED:(0)} nfs/arch-paul.home@home for krbtgt/hom@home, Server not found in Kerberos database

This is the principal list on the kerberos server:

kadmin.local:  list_principals
K/M@home
host/arch-paul.home@home
host/sup.home@home
kadmin/admin@home
kadmin/changepw@home
krbtgt/home@home
nfs/arch-paul.home@home
nfs/sup.home@home
paul@home

Not sure if it is the issue, but `krbtgt/hom@home` seems like a typo, but I'm not sure where that typo is.

Last edited by pvdl1984 (2021-12-28 17:01:13)

seth · 2021-12-28 17:20:22

"klist" on the client?
The cache should™ be somewhere in /tmp/krb5*

pvdl1984 · 2021-12-28 17:36:11

I also found the second issue: which was a typo in the kerberos configuration ([domain_realm] had indeed the misspelled hom instead of home) .

The last issue I'm having that I can't use the mount as a non root user, e.g.:

$ ls -lha /mnt                                                                                                                                                                                              
ls: cannot open directory '/mnt': Stale file handle

$ sudo ls -lha /mnt                                                                                                                                                                                              
total 0
drwxr-xr-x 1 paul users   2 Dec 27 17:12 .
drwxr-xr-x 1 root root  172 Dec  9 13:10 ..
-rw-r--r-- 1 paul users   0 Dec 27 17:12 p

seth · 2021-12-28 22:16:34

https://unix.stackexchange.com/question … not-umount

pvdl1984 · 2021-12-29 10:28:58

seth wrote:

https://unix.stackexchange.com/question … not-umount

This seems to be unrelated. I have tried all the troubleshooting here, and on other search results for "nfs stale ...". None of the troubleshooting on those pages help, and none match my error. They seem to be all related to mounts being left mounted for a while, then the nfs server restarts, a file get's deleted or anything like that.

This is literrally:
- step one: mount
- step two: ls -> fail
- step three: sudo ls -> ok

Basically for some reason NFS is not working if I'm not root, even while idmapping is working, and the permissions and owners on the server match my user on the client.

pvdl1984 · 2021-12-29 11:11:38

It is working without Kerberos. which got me to try to do `kinit` and then try to list. This seems to all work. Does this mean I have to `kinit` every time I want to use the mount if using Kerberos?

pvdl1984 · 2021-12-29 11:37:33

I think I resolved all those issues now. I need to create the user keytab file (instead of a global one owned by root according to the wiki). (Finding the location by using krb5-config --defcktname) Thanks for the help @seth!

Last edited by pvdl1984 (2021-12-29 11:37:52)

pvdl1984 · 2021-12-29 11:43:28

Maybe not. Now I can't write to the NFS share. While I should be able to according to the permissions (unless I'm tired and started to misunderstand permissions as well).

$ ls -lha /mnt/personal 
total 0
drwxr-xr-x 1 paul users  0 Dec 29 12:39 .
drwxr-xr-x 1 root root  58 Dec 26 12:14 ..
$ whoami                                                                                                                                                                                                          
paul
$ touch /mnt/personal/test
touch: cannot touch '/mnt/personal/test': Permission denied

user id's match on both machines, and I enabled idmapping (at least I think so, unless I misunderstood the instructions).

Although I would assume reading wouldn't work either if idmap is failing, but when checking for idmap, it doesn't work on the server.
Server:

$ sudo dmesg | grep id_resolver

Client:

$ sudo dmesg | grep id_resolver                                                                                                                                             
[   28.822965] NFS: Registering the id_resolver key type
[   28.822977] Key type id_resolver registered

Unfortunately the wiki doesn't explain how to resolve this. I configured this the same on both client and server:

$ cat  /etc/modprobe.d/nfsd.conf
options nfs nfs4_disable_idmapping=0
options nfsd nfs4_disable_idmapping=0

$ cat /sys/module/nfs/parameters/nfs4_disable_idmapping  # Client
N
$ cat /sys/module/nfsd/parameters/nfs4_disable_idmapping  # Server
N 

$ cat /etc/idmapd.conf                                                                                                                                                                          
[General]
#Verbosity = 0
# The following should be set to the local NFSv4 domain name
# The default is the host's DNS domain name.
Domain = home
...

On the other end, this is what happens after mounting and listing the share. If I run the following on the client:

sudo nfsidmap -l                                                                                                                                                                                                
4 .id_resolver keys found:
  gid:users@home
  uid:paul@home
  gid:root@home
  uid:root@home

Last edited by pvdl1984 (2021-12-29 12:32:19)

pvdl1984 · 2021-12-29 16:02:18

- with rpcdebug it looks like everything is working (lthough I'm not sure how to see nfsidmap information). i get the expected uid, gid, permission mask
- ls shows me the details I'm expected
- but while my username and uid are both matching the owner, and the same for the group, all my access is being asses as "other"
- programs who gives you feedback (like vim and gui filemanagers), say a file or folder is readonly to me. But when I ask the same program to display me the permissions, I should be owner and have write access.

seth · 2021-12-30 16:21:18

https://www.mail-archive.com/dng@lists. … 29294.html
https://serverfault.com/questions/10883 … t-not-root

And what is the server? Because there once upon a time not so long ago, there was a 32bit related bug:
https://bugs.archlinux32.org/index.php? … task_id=92
https://forums.gentoo.org/viewtopic-t-1 … art-0.html (seem to be the same issue)

pvdl1984 · 2021-12-31 13:25:49

I finally figured out the issue, by digging into the kerberos authentication. The documentation on how the combination of nfs, nfsidmap and kerberos work together is still a mystery though.

The main thing was this: https://web.mit.edu/kerberos/krb5-lates … horization
My client was authenticated with the principal `nfs/arch-paul.home@home`, as most tutorials, wiki's and docs instruct you to do. Which doesn't translate to any user (at least in the default configuration). When I made sure I was authenticating with my paul@home principal, it was actually mapped to the paul user on the NFS server. At which point I actually had the owner permissions as I expected.

I got distracted by nfsidmap, which I thought was actually taking care of translating the local user to the remote user (which didn't sound very secure), but I think it's only used for display purposes as far as I understand.

Arch Linux

#1 2021-12-27 16:39:27

NFS & Kerberos not working: export does not exist

#2 2021-12-27 21:16:16

Re: NFS & Kerberos not working: export does not exist

#3 2021-12-28 10:30:51

Re: NFS & Kerberos not working: export does not exist

#4 2021-12-28 13:35:20

Re: NFS & Kerberos not working: export does not exist

#5 2021-12-28 15:33:08

Re: NFS & Kerberos not working: export does not exist

#6 2021-12-28 15:50:18

Re: NFS & Kerberos not working: export does not exist

#7 2021-12-28 16:21:02

Re: NFS & Kerberos not working: export does not exist

#8 2021-12-28 16:46:07

Re: NFS & Kerberos not working: export does not exist

#9 2021-12-28 16:59:58

Re: NFS & Kerberos not working: export does not exist

#10 2021-12-28 17:20:22

Re: NFS & Kerberos not working: export does not exist

#11 2021-12-28 17:36:11

Re: NFS & Kerberos not working: export does not exist

#12 2021-12-28 22:16:34

Re: NFS & Kerberos not working: export does not exist

#13 2021-12-29 10:28:58

Re: NFS & Kerberos not working: export does not exist

#14 2021-12-29 11:11:38

Re: NFS & Kerberos not working: export does not exist

#15 2021-12-29 11:37:33

Re: NFS & Kerberos not working: export does not exist

#16 2021-12-29 11:43:28

Re: NFS & Kerberos not working: export does not exist

#17 2021-12-29 16:02:18

Re: NFS & Kerberos not working: export does not exist

#18 2021-12-30 16:21:18

Re: NFS & Kerberos not working: export does not exist

#19 2021-12-31 13:25:49

Re: NFS & Kerberos not working: export does not exist

Board footer