NFS server: can't access files

bbaserdem · 2020-05-31 04:08:00

I have been trying to share my files on my laptop (for Kodi) in my network. I followed the NFS wiki page. I can see my folders (both showmount and kodi) as advertised but can't access anything inside of them. I have the following system layout;

My /etc/exports is;

# Share NFS to host media files
/srv/nfs                    192.168.1.0/24(ro,fsid=0,no_subtree_check)
/srv/nfs/music              192.168.1.0/24(ro,no_subtree_check,insecure)
/srv/nfs/videos             192.168.1.0/24(ro,no_subtree_check,insecure)
/srv/nfs/pictures           192.168.1.0/24(ro,no_subtree_check,insecure)
/srv/nfs/music-archive      192.168.1.0/24(ro,no_subtree_check,insecure)
/srv/nfs/videos-archive     192.168.1.0/24(ro,no_subtree_check,insecure)
/srv/nfs/pictures-archive   192.168.1.0/24(ro,no_subtree_check,insecure)

Each of the directories in /srv/nfs are bind mounts to directories that are 755 permission with uid/guid of 1000.

Result of sudo exportfs -v

/srv/nfs      	                    192.168.1.0/24(sync,wdelay,hide,no_subtree_check,fsid=0,sec=sys,ro,secure,root_squash,no_all_squash)
/srv/nfs/music	            192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,insecure,root_squash,no_all_squash)
/srv/nfs/videos                  192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,insecure,root_squash,no_all_squash)
/srv/nfs/pictures               192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,insecure,root_squash,no_all_squash)
/srv/nfs/music-archive     192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,insecure,root_squash,no_all_squash)
/srv/nfs/videos-archive    192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,insecure,root_squash,no_all_squash)
/srv/nfs/pictures-archive 192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,insecure,root_squash,no_all_squash)

My ports are open (I'm using firewalld; explicitly opened the ports) Output of rpcinfo -p (This is the identical result when i run rpcinfo using my server hostname on another computer)

    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100024    1   udp  45946  status
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100024    1   tcp  47539  status
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049  nfs_acl
    100021    1   udp  55535  nlockmgr
    100021    3   udp  55535  nlockmgr
    100021    4   udp  55535  nlockmgr
    100021    1   tcp  41497  nlockmgr
    100021    3   tcp  41497  nlockmgr
    100021    4   tcp  41497  nlockmgr

Running showmount from both another computer (using the proper hostname) and localhost gives me the advertised directories

$ showmount -e localhost
Export list for localhost:
/srv/nfs/pictures-archive 192.168.1.0/24
/srv/nfs/videos-archive   192.168.1.0/24
/srv/nfs/music-archive    192.168.1.0/24
/srv/nfs/pictures         192.168.1.0/24
/srv/nfs/videos           192.168.1.0/24
/srv/nfs/music            192.168.1.0/24
/srv/nfs                  192.168.1.0/24

Here is trying to mount NFS directories from the server

# mount -vvv sbp-homestation:/music /mnt
mount.nfs: timeout set for Sun May 31 00:00:30 2020
mount.nfs: trying text-based options 'vers=4.2,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=127.0.1.1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 127.0.1.1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 127.0.1.1 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting sbp-homestation:/music

And here is trying to mount from a client

# mount -vvv sbp-homestation:/music /mnt
mount.nfs: timeout set for Sun May 31 00:00:30 2020
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'addr=192.168.1.8'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.8 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
<Repeats until timeout>

I am sure it's not the firewall; since I can turn off firewalld and i will get the same error. (And I opened all the ports) I was able to add files on kodi yesterday. I installed avahi today, because I have a non-static ip in this network; and wanted to do it using zeroconf instead of hardcoded ip addresses. However, now nothing works. I can't even access NFS from the server machine itself. Can anyone offer any insight?

seth · 2020-05-31 06:34:50

man exports, check for "crossmnt"

bbaserdem · 2020-05-31 15:16:13

So I did read through the man page again. I don't know if crossmnt should apply to me; since the files I'm broadcasting are the bind points themselves; there is no further fs shenaningans going on; my lsblk and fstab is as follows

$ lsblk
NAME                     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                        8:0    0 931.5G  0 disk  
├─sda1                     8:1    0   200G  0 part  /media/windows
└─sda2                     8:2    0 731.5G  0 part  
  └─Homestation-Archive  254:5    0 731.5G  0 crypt /media/archive
nvme0n1                  259:0    0 931.5G  0 disk  
├─nvme0n1p1              259:1    0   550M  0 part  /boot/efi
└─nvme0n1p2              259:2    0   931G  0 part  
  └─cryptlvm             254:0    0   931G  0 crypt 
    ├─Homestation-Arch   254:1    0    45G  0 lvm   /var/lib/docker/btrfs
    ├─Homestation-Home   254:2    0   799G  0 lvm   /home
    ├─Homestation-Gentoo 254:3    0    45G  0 lvm   /media/gentoo
    └─Homestation-Opt    254:4    0    42G  0 lvm   /opt
$ cat /etc/fstab
/dev/mapper/Homestation-Arch    /               btrfs   rw,noatime,nodiratime,compress=lzo,ssd,space_cache,subvol=@root                                             0       0
/dev/mapper/Homestation-Arch    /.snapshots     btrfs   rw,noatime,nodiratime,compress=lzo,ssd,space_cache,subvol=@snapshots                                        0       0
/dev/mapper/Homestation-Opt     /opt            ext4    defaults                                                                                                    0       0
UUID=7653-3DB8                  /boot/efi       vfat    rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro   0       2
/dev/mapper/Homestation-Home    /home           xfs     rw,relatime,attr2,inode64,noquota                                                                           0       2
/dev/mapper/Homestation-Archive /media/archive  ext4    defaults                                                                                                    0       2
/swap/swapfile                  none            swap    defaults                                                                                                    0       0
# Windows
LABEL=Windows10                 /media/windows  ntfs-3g defaults,nls=utf8,umask=000,dmask=027,fmask=137,uid=1000,gid=1000,windows_names                             0       0
# Gentoo
/dev/mapper/Homestation-Gentoo  /media/gentoo               btrfs   rw,noatime,nodiratime,compress=lzo,ssd,space_cache,subvol=@root                                 0       0
# NFS for Kodi
/home/sbp/Music         /srv/nfs/music              none    bind    0   0
/home/sbp/Videos        /srv/nfs/videos             none    bind    0   0
/home/sbp/Pictures      /srv/nfs/pictures           none    bind    0   0
/media/archive/Music    /srv/nfs/music-archive      none    bind    0   0
/media/archive/Videos   /srv/nfs/videos-archive     none    bind    0   0
/media/archive/Pictures /srv/nfs/pictures-archive   none    bind    0   0

I did try the crossmnt option, and the nohide option (and ran exportfs afterwards of course); but i still can't mount the directories.

Last edited by bbaserdem (2020-05-31 15:18:54)

seth · 2020-05-31 19:14:35

Can you try to get the bind problem out of the way and only export one path that's on the root filesystem with files in it that are on the filesystem as well?

Also, for clarification:

mount -vvv sbp-homestation:/music /mnt

"/music" isn't exported, but eg. "/srv/nfs/music" is. Are you simply trying to mount the wrong remote paths?

bbaserdem · 2020-06-01 05:33:56

I'm sure. This mount path was working 3 days ago when I first configured it. (I'm pulling the commands from my zsh history). After a reboot; I tried setting up avahi on top of it. Here is what happens when I try mounting using the full path from the localhost of the nfs server;

$ sudo mount -vvv sbp-homestation:/srv/nfs/music /mnt
mount.nfs: timeout set for Mon Jun  1 01:28:43 2020
mount.nfs: trying text-based options 'vers=4.2,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,addr=127.0.1.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=127.0.1.1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 127.0.1.1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 127.0.1.1 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting sbp-homestation:/srv/nfs/music

My ultimate goal is to access my files using Kodi on my firestick in my network. So I'm not concerned with being able to mount that much; that's my test of if NFS is working. Currently; using Kodi on any of my (homestation, laptop, firestick, phone. I have already setup mariadb server on the homestation device to act as a central db and it works) devices; I can see my server both in the NFS and the Zeroconf directories. (I see all the directories that my configuration is sharing.) It's just that I cannot actually go into those directories; or access any files in them.

Trying to mount the filesystem is just a test for me. But in my experience; if I cannot mount using nfs; there is no way any other program is going to have access anyway.

Last edited by bbaserdem (2020-06-01 05:35:24)

seth · 2020-06-01 10:56:28

was working 3 days ago … tried setting up avahi on top

Is that the only change you made?
Did you also run an update?
How exactly did you try to setup avahi? Did you? What if you unroll those changes?
Did you restrict the access in nfs.conf?

CoudUr · 2020-06-01 11:01:47

Hi,

I don't know much about NFS or Avahi but to me it looks like the latter is the one you should look into especially if you say that your configuration was working fine before it was installed.
I think the warning messages "Permission denied" and "access denied by server" imply something is missing, is wrongly configured, or has changed. I don't know if you already did it but may be reset your /etc/hosts as it was before Avahi installation since it doesn't serve any purpose anymore.
Looking deeper at the logs system from the server and clients by using journalctl and dmesg will hopefully help you more than the only NFS specific tools.

bbaserdem · 2020-06-02 01:26:38

Thank you for the suggestions. I'm facepalming at not thinking about removing

* I saw nothing out of ordinary in the systemd unit statuses. NSS server forks so nothing useful about that. Nothing indicating an error on avahi-daemon. I could not locate nss or avahi logs.
* I did try creating a path /srv/nfs/testing; creating a file as /srv/nfs/testing/test; (changing owner to my user; UID 1000) and adding the entry to /etc/exports. The file I can see again; but still can't mount it.
* I did stop avahi-daemon socket and service; and undid the mdns line from /etc/nsswitch.conf. I'm getting the same error messages. My nsswitch config (before and after) is as follows

# Name Service Switch configuration file.
# See nsswitch.conf(5) for details.

passwd: files mymachines systemd
group: files mymachines systemd
shadow: files

publickey: files

#hosts: files mymachines myhostname mdns resolve dns
hosts: files mymachines myhostname resolve dns
networks: files

protocols: files
services: files
ethers: files
rpc: files

netgroup: files

* I used my previous /etc/exports that previously worked; which is

# Managed by sbp

# Share NFS to host media files
/srv/nfs                192.168.1.0/24(rw,fsid=0,no_subtree_check)
/srv/nfs/music          192.168.1.0/24(rw,no_subtree_check,insecure,no_root_squash)
/srv/nfs/videos         192.168.1.0/24(rw,no_subtree_check,insecure,no_root_squash)
/srv/nfs/music-archive  192.168.1.0/24(rw,no_subtree_check,insecure,no_root_squash)
/srv/nfs/videos-archive 192.168.1.0/24(rw,no_subtree_check,insecure,no_root_squash)

And still nothing
The way I configured avahi was I have a file for each of the directories I wanted host; which goes like this;

$ cat /etc/avahi/services/nfs_kodi_music.service
<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">NFS Music Share on %h</name>
  <service>
    <type>_nfs._tcp</type>
    <port>2049</port>
    <txt-record>path=/srv/nfs/music</txt-record>
  </service>
</service-group>

I'm going to restart with avahi disabled and with my old config to see if that changes anything.

EDIT: Nothing has changed. I don't even understand what is going on with regards to missing permission. I am using mount with sudo; my ports are open, i can see that my directories are configured for both avahi and nfs. I'm at a full loss at this one.

EDIT2: I decided to set everything from scratch again. I disabled all services and sockets, closed all the opened ports, removed bind mounts from fstab, removed all config files and refreshed everything again. So far; I am following the archwiki; and I notice if I don't open ports besides tcp 2049; i can't see anything on showmount -e; both from the server and clients. Also; can't see the host in kodi; besides the localhost (but only on the interface 192.168.122.1 which is a bit weird; since I could previously see myself on the kodi page as either 127.0.0.1 as localhost; or my local ip address which is 192.168.1.9/24)

Last edited by bbaserdem (2020-06-02 02:49:10)

bbaserdem · 2020-06-02 03:50:17

So; I removed all configuration and restarted. Here are the exact steps I taken;

* I create a directory /srv/nfs/media; and subdirectories there as bind mounts to my home folder data location
* I put the info in fstab; and mounted them using 'mount -a' to check that everything is working ok.
* I created the following /etc/exports; and then enabled and started nfs-server.service, and ran exportfs -arv

/srv/nfs/       192.168.1.0/24(rw,sync,crossmnt,fsid=0)
/srv/nfs/media  192.168.1.0/24(ro,all_squash,insecure,crossmnt)

* I went to firewalld; and set my current zone to allow nfs; which I learned from /usr/lib/firewalld/services/nfs.xml that it opens tcp port 2049
* At this point; i tried some diagnostic stuff through both the server and a client machine.
** On server; showmount -e localhost (or sbp-homestation) DOES list my two directories as available mounts. I CANNOT mount these directories by using 'mount localhost:/ /mnt', 'mount sbp-homestation:/', 'mount localhost:/srv/nfs/ /mnt', 'mount sbp-homestation:/srv/nfs'.
** On client; showmount -e sbp-homestation DOES NOT list anything; i have the error message 'clnt_create: RPC: Unable to receive' I CAN mount using 'mount sbp-homestation:/ /mnt' and explore directory. I CANNOT mount using 'mount sbp-homestation:/srv/nfs /mnt' with the error 'No such file or directory'.
* After this; I decided to setup avahi on top;
* I opened the mdns service in firewalld; which is the udp 5353 port suggested by archwiki.
* I added mdns before the resolve line in /etc/nsswitch.conf
* I enabled and started avahi-daemon.service on both the host and the client
* When I run avahi-browse --all --resolve --terminate | grep sbp ; i can see the nfs service.
* I still cannot mount, using nfs, on the server; (I don't have any localhost lines for this; so I'm fine with it) but I can mount it on the client side. I actually don't know how to test if mount is working using zeroconf.
* Kodi cannot detect my server on the NFS menu. (It can; but through the local IP address only on the server) My zeroconf NFS server is seen by kodi; but I cannot traverse directories.

On kodi log; i see these line when I try to browse using the zeroconf, and try to enter the nfs menu;

* ERROR: Unable to lookup host: '<IPv6 address>'
* ERROR: GetDirectory - Error getting nfs://[<IPv6 address>]:2049/srv/nfs/media/
* DEBUG: CZeroconfBrowserAvahi::resolveCallback resolved service 'NFS Media Share on sbp-homestation' of type '_nfs._tcp' in domain 'local':

I installed thunar, gvfs and gvfs-nfs; because I saw gvfs as a target for zeroconf ntfs drives. I'm getting a "Failed to open NFS Media Share on sbp-homestation ; mount point does not exist" error.

Last edited by bbaserdem (2020-06-02 04:39:17)

seth · 2020-06-02 06:35:20

Resolving sbp-homestation will probably rely on avahi (unless you've a differen DNS system at play, eg. dnsmasq in your local router etc.) so you'll have to use the IPs w/o.
At this point I'm not sure whether "since I can turn off firewalld" did what you expected and didn't just leave the established nftables behind - notably since you're using the insecure option what allows for random client ports.

Disable the firewalld and reboot (or stop it and flush all nft rules, check the remaining rules: https://wiki.archlinux.org/index.php/Nftables )
Also keep avahi disabled and operate on the IPs. We best get a working baseline to incrementally add complexity to see at which point you break it.

bbaserdem · 2020-06-02 21:12:24

That was the case. Disabling everything but nfs-server and then rebooting; i was able to view and browse the directory. Then starting avahi on top; i was able to browse the directory.

My firewall configuration was not correct as I though it was. I noticed that two days ago; when everything was working; my wifi interface was not setup properly by firewalld; and fixed that. I was probably lost at that point; but thought things were working because kodi was trying to get metadata.

When I find all the ports that are utilized to make things work; I'll edit the title with [SOLVED].

EDIT: I closed all open ports; and am trying to open them one by one;

* No firewall; everything fine and dandy.
* Opening upd 2049 (for firewalld; default nfs3 service has both tcp and upd for port 2049) allows me to see and traverse my server on 192.168.1.8/24 using kodi.
* Opening mdns service (udp port 5353 for 224.0.0.251 which I think something like local machines) allows me to see my nfs service on avahi; but I can't traverse the directory.
* Opening 111 or 20048 does nothing.

EDIT2: I'm a bit burned out from trying to resolve this issue. I think my issue with the firewall is solved; and the rest is just kodi and other stuff issue. I realized that avahi does not fix my non-static ip problem (the couple times I could add files through kodi; the zeroconf address just turns into my currently leased ip address of the server) so I don't need-need avahi for my purposes. And as long as I don't have a way of setting a static ip for my server (I don't have access to the router) I really dont get any benefits from avahi. The behavior being super inconsistent; and managing to mount even with the firewall, tells me that my nfs setup is proper now. I think the rest is full-on Kodi; because thats where I get a lot of freezes trying to traverse files etc. I'm giving up on making kodi work for now; but i'll be listening in case i can be retroactively corrected.

Last edited by bbaserdem (2020-06-03 05:11:21)

Arch Linux

#1 2020-05-31 04:08:00

NFS server: can't access files

#2 2020-05-31 06:34:50

Re: NFS server: can't access files

#3 2020-05-31 15:16:13

Re: NFS server: can't access files

#4 2020-05-31 19:14:35

Re: NFS server: can't access files

#5 2020-06-01 05:33:56

Re: NFS server: can't access files

#6 2020-06-01 10:56:28

Re: NFS server: can't access files

#7 2020-06-01 11:01:47

Re: NFS server: can't access files

#8 2020-06-02 01:26:38

Re: NFS server: can't access files

#9 2020-06-02 03:50:17

Re: NFS server: can't access files

#10 2020-06-02 06:35:20

Re: NFS server: can't access files

#11 2020-06-02 21:12:24

Re: NFS server: can't access files

Board footer