You are not logged in.

#1 2021-11-24 23:10:11

dot1910
Member
Registered: 2021-11-24
Posts: 6

[solved] NFS client not working

I have been using NFS4 with AutoFS for sometimes in my server and desktop (NFS clients) accessing Synology NAS 6.2 (with NFSv4 server) sharing `/volume1/files`. 

Recently the setup no longer work. I noticed that I can see the `/net/nas/volume1/files` but couldn't `ls` or access the shared NFS folder.

I am getting this error on manual mount.

$ sudo mount -t nfs4 -v -v -v nas:/volume1/files /mnt/nas-files
mount.nfs4: trying text-based options 'vers=4.0,addr=10.42.0.251,clientaddr=10.42.0.100'
mount.nfs4: mount(2): Connection timed out
mount.nfs4: Connection timed out

This is the relevant part of

sudo journalctl -fl

(using rpcdebug with nfsd, nfs, nlm, rpc module)

Nov 25 18:21:01 deltax kernel: NFS:   parsing nfs mount option 'source'
Nov 25 18:21:01 deltax kernel: NFS:   parsing nfs mount option 'nfsvers'
Nov 25 18:21:01 deltax kernel: NFS:   parsing nfs mount option 'addr'
Nov 25 18:21:01 deltax kernel: NFS:   parsing nfs mount option 'clientaddr'
Nov 25 18:21:01 deltax kernel: NFS: MNTPATH: '/volume1/files'
Nov 25 18:21:01 deltax kernel: --> nfs4_try_get_tree()
Nov 25 18:21:01 deltax kernel: NFS: get client cookie (0x00000000670a8ad4/0x0000000023984d7e)
Nov 25 18:21:01 deltax kernel: RPC:       set up xprt to 10.42.0.251 (port 2049) via tcp
Nov 25 18:21:01 deltax kernel: RPC:       Couldn't create auth handle (flavor 390004)
Nov 25 18:21:01 deltax kernel: RPC:        destroy backchannel transport
Nov 25 18:21:01 deltax kernel: RPC:        backchannel list empty= true
Nov 25 18:21:01 deltax kernel: RPC:       xs_destroy xprt 000000007867c1f2
Nov 25 18:21:01 deltax kernel: RPC:       xs_close xprt 000000007867c1f2
Nov 25 18:21:01 deltax kernel: nfs_create_rpc_client: cannot create RPC client. Error = -22
Nov 25 18:21:01 deltax kernel: RPC:       set up xprt to 10.42.0.251 (port 2049) via tcp
Nov 25 18:21:01 deltax kernel: RPC:       xs_connect scheduled xprt 00000000797591fa
Nov 25 18:21:01 deltax kernel: RPC:       xs_bind 0.0.0.0:675: ok (0)
Nov 25 18:21:01 deltax kernel: RPC:       worker connecting xprt 00000000797591fa via tcp to 10.42.0.251 (port 2049)
Nov 25 18:21:01 deltax kernel: RPC:       00000000797591fa connect status 115 connected 0 sock state 2

Nov 25 18:24:02 deltax kernel: nfs_create_rpc_client: cannot create RPC client. Error = -110
Nov 25 18:24:02 deltax kernel: NFS: releasing client cookie (0x00000000670a8ad4/0x0000000023984d7e)
Nov 25 18:24:02 deltax kernel: NFS4: Couldn't follow remote path
Nov 25 18:24:02 deltax kernel: <-- nfs4_try_get_tree() = -110 [error]
Nov 25 18:24:02 deltax kernel: RPC:        destroy backchannel transport
Nov 25 18:24:02 deltax kernel: RPC:        backchannel list empty= true
Nov 25 18:24:02 deltax kernel: RPC:       xs_destroy xprt 00000000797591fa
Nov 25 18:24:02 deltax kernel: RPC:       xs_close xprt 00000000797591fa
Nov 25 18:24:02 deltax kernel: RPC:       xs_error_report client 00000000797591fa, error=104...
Nov 25 18:24:02 deltax kernel: RPC:       xs_tcp_state_change client 00000000797591fa...
Nov 25 18:24:02 deltax kernel: RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 0

I have tried with

- the following kernel 5.15, 5.10, 4.19
- with nfs-utils 2.5.4 and 2.5.3
- clean server setup on server and client on desktop

and still no luck.

- both server and desktop can ping the NAS (and vice versa)
- there are no firewall setup on desktop and NAS, server is setup to allow all in local subnet
- `rpcinfo -p nas` (running on server and desktop) output is correct
- `showmount -e nas` shows `/volume1/files  *`

$ rpcinfo -p nas
program vers proto   port  service
100000    4   tcp    111  portmapper
100000    3   tcp    111  portmapper
100000    2   tcp    111  portmapper
100000    4   udp    111  portmapper
100000    3   udp    111  portmapper
100000    2   udp    111  portmapper
100005    1   udp    892  mountd
100005    1   tcp    892  mountd
100005    2   udp    892  mountd
100005    2   tcp    892  mountd
100005    3   udp    892  mountd
100005    3   tcp    892  mountd
100003    2   tcp   2049  nfs
100003    3   tcp   2049  nfs
100003    4   tcp   2049  nfs
100003    2   udp   2049  nfs
100003    3   udp   2049  nfs
100021    1   udp  57115  nlockmgr
100021    3   udp  57115  nlockmgr
100021    4   udp  57115  nlockmgr
100021    1   tcp  58397  nlockmgr
100021    3   tcp  58397  nlockmgr
100021    4   tcp  58397  nlockmgr
100024    1   udp  51627  status
100024    1   tcp  52924  status

I have also read
- NFS Troubleshooting Procedures
- Archlinux Wiki NFS/Troubleshooting: which claims it is outdated for NFSv4

Anybody having this kind of problem? Did you solve it and how did you do it?

EDIT 1: correct rpcdebug output (from Arch installation, prev was Arch32), and format with code, inserted output of nfs-ls

$ sudo nfs-ls nfs://nas/volumes/files
[sudo] password for user:
Failed to mount nfs share : nfs_service failed

Last edited by dot1910 (2021-11-30 21:32:17)

Offline

#2 2021-11-25 09:13:16

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [solved] NFS client not working

1. please use code tags, not quote tags
2. don't copy out of the pager, ther journal paste is messed up (eg. redirect it into a file instead) - the area around error 512 looks interesting, but is also malformed.
3. does nfs-ls work?

Offline

#3 2021-11-25 14:20:18

dot1910
Member
Registered: 2021-11-24
Posts: 6

Re: [solved] NFS client not working

I have corrected rpcdebug output (from Arch installation, prev was Arch32), and format with code, inserted output of nfs-ls which is

$ sudo nfs-ls nfs://nas/volumes/files
[sudo] password for user:
Failed to mount nfs share : nfs_service failed

during which rpcdebug does not produce any output.

Mount with vers 3

$ sudo mount -vvv -t nfs -o nfsvers=3
mount.nfs4: timeout set for Thu Nov 25 20:47:26 2021
mount.nfs4: trying text-based options 'nfsvers=3,addr=10.42.0.251'
mount.nfs4: prog 100003, trying vers=3, prot=6
mount.nfs4: trying 10.42.0.251 prog 100003 vers 3 prot TCP port 2049
mount.nfs4: prog 100005, trying vers=3, prot=17
mount.nfs4: trying 10.42.0.251 prog 100005 vers 3 prot UDP port 892

log shows

Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'source'
Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'addr'
Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'vers'
Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'proto'
Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'mountvers'
Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'mountproto'
Nov 25 20:45:26 deltax kernel: NFS:   parsing nfs mount option 'mountport'
Nov 25 20:45:26 deltax kernel: NFS: MNTPATH: '/volume1/files'
Nov 25 20:45:26 deltax kernel: NFS: sending MNT request for sigma:/volume1/files
Nov 25 20:45:26 deltax kernel: RPC:       set up xprt to 10.42.0.251 (port 892) via udp
Nov 25 20:45:26 deltax kernel: RPC:       xs_connect scheduled xprt 00000000ee30d82b
Nov 25 20:45:26 deltax kernel: RPC:       xs_bind 0.0.0.0:810: ok (0)
Nov 25 20:45:26 deltax kernel: RPC:       worker connecting xprt 00000000ee30d82b via udp to 10.42.0.251 (port 892)
Nov 25 20:45:26 deltax kernel: RPC:       xs_udp_send_request(40) = 0
Nov 25 20:45:26 deltax kernel: RPC:       xs_data_ready...
Nov 25 20:45:26 deltax kernel: RPC:       cong 256, cwnd was 256, now 512
Nov 25 20:45:26 deltax kernel: RPC:       xs_udp_send_request(92) = 0
Nov 25 20:45:26 deltax kernel: RPC:       xs_data_ready...
Nov 25 20:45:26 deltax kernel: RPC:       cong 256, cwnd was 512, now 512
Nov 25 20:45:26 deltax kernel: NFS: received 1 auth flavors
Nov 25 20:45:26 deltax kernel: NFS:   auth flavor[0]: 1
Nov 25 20:45:26 deltax kernel: NFS: MNT request succeeded
Nov 25 20:45:26 deltax kernel: NFS: attempting to use auth flavor 1
Nov 25 20:45:26 deltax kernel: NFS: get client cookie (0x00000000559f09b2/0x000000007bd41024)
Nov 25 20:45:26 deltax kernel: RPC:       set up xprt to 10.42.0.251 (autobind) via tcp
Nov 25 20:45:26 deltax kernel: RPC:       set up xprt to 10.42.0.251 (port 111) via tcp
Nov 25 20:45:26 deltax kernel: RPC:       xs_destroy xprt 00000000ee30d82b
Nov 25 20:45:26 deltax kernel: RPC:       xs_close xprt 00000000ee30d82b
Nov 25 20:45:26 deltax kernel: RPC:       xs_connect scheduled xprt 00000000dff2b871
Nov 25 20:45:26 deltax kernel: RPC:       worker connecting xprt 00000000dff2b871 via tcp to 10.42.0.251 (port 111)
Nov 25 20:45:26 deltax kernel: RPC:       00000000dff2b871 connect status 115 connected 0 sock state 2
Nov 25 20:45:26 deltax kernel: RPC:       xs_tcp_state_change client 00000000dff2b871...
Nov 25 20:45:26 deltax kernel: RPC:       state 1 conn 0 dead 0 zapped 1 sk_shutdown 0
Nov 25 20:45:26 deltax kernel: RPC:       xs_tcp_send_request(84) = 0
Nov 25 20:45:26 deltax kernel: RPC:       xs_data_ready...
Nov 25 20:45:26 deltax kernel: RPC:       setting port for xprt 00000000d9186d99 to 2049
Nov 25 20:45:26 deltax kernel: RPC:        destroy backchannel transport
Nov 25 20:45:26 deltax kernel: RPC:        backchannel list empty= true
Nov 25 20:45:26 deltax kernel: RPC:       xs_destroy xprt 00000000dff2b871
Nov 25 20:45:26 deltax kernel: RPC:       xs_close xprt 00000000dff2b871
Nov 25 20:45:26 deltax kernel: RPC:       xs_tcp_state_change client 00000000dff2b871...
Nov 25 20:45:26 deltax kernel: RPC:       state 4 conn 1 dead 0 zapped 1 sk_shutdown 3
Nov 25 20:45:26 deltax kernel: RPC:       xs_connect scheduled xprt 00000000d9186d99
Nov 25 20:45:26 deltax kernel: RPC:       xs_bind 0.0.0.0:905: ok (0)
Nov 25 20:45:26 deltax kernel: RPC:       worker connecting xprt 00000000d9186d99 via tcp to 10.42.0.251 (port 2049)
Nov 25 20:45:26 deltax kernel: RPC:       00000000d9186d99 connect status 115 connected 0 sock state 2

Nov 25 20:47:36 deltax kernel: RPC:       xs_error_report client 00000000d9186d99, error=110...
Nov 25 20:47:36 deltax kernel: RPC:       xs_tcp_state_change client 00000000d9186d99...
Nov 25 20:47:36 deltax kernel: RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
Nov 25 20:47:36 deltax kernel: RPC:       xs_tcp_state_change client 00000000d9186d99...
Nov 25 20:47:36 deltax kernel: RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
Nov 25 20:47:36 deltax kernel: RPC:       xs_connect scheduled xprt 00000000d9186d99
Nov 25 20:47:36 deltax kernel: RPC:       xs_bind 0.0.0.0:905: ok (0)
Nov 25 20:47:36 deltax kernel: RPC:       worker connecting xprt 00000000d9186d99 via tcp to 10.42.0.251 (port 2049)
Nov 25 20:47:36 deltax kernel: RPC:       00000000d9186d99 connect status 115 connected 0 sock state 2

then mount command ends with

mount.nfs4: mount(2): Connection timed out
mount.nfs4: Connection timed out

Offline

#4 2021-11-25 16:04:13

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [solved] NFS client not working

So you've timeouts from arch and arch32 connecting the same synology NAS?
Wrt the latter: have you tried turning it off and on again?

Offline

#5 2021-11-25 23:32:15

dot1910
Member
Registered: 2021-11-24
Posts: 6

Re: [solved] NFS client not working

Yes, I have tried basic things like
- reboot,
- power off/on,
- network cable plug/unplug.

The problem is there for some weeks now. This is the second time I try to troubleshoot and make it work.

Couldn't find much info on those RPC and NFS errors.

Offline

#6 2021-11-26 07:17:45

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [solved] NFS client not working

Does this correspond w/ an update to the NAS?
Can you impact/read out its NFS configuration? Logs?

W/ nfs3 you could try "nolock" and "proto=udp,mountproto=udp" and w/ either 3 or 4 "sec=none"
You could also attempt a connection from an entirely different sw stack (ie. some - older - live distro)…

Offline

#7 2021-11-27 11:36:42

dot1910
Member
Registered: 2021-11-24
Posts: 6

Re: [solved] NFS client not working

Yes!! The options "proto=udp,mountproto=udp" works for me. You are a life saver.

Still the NFSv4 is no luck.

- NAS is set to auto update but no major version for a long time (DSM 6.2).
- there are the output I get from showmount, rpcinfo, no nfs-ls, no nfs.conf

admin@nas:/$ rpcinfo
program version netid     address                service    owner
100000    4    tcp6      ::.0.111               -          superuser
100000    3    tcp6      ::.0.111               -          superuser
100000    4    udp6      ::.0.111               -          superuser
100000    3    udp6      ::.0.111               -          superuser
100000    4    tcp       0.0.0.0.0.111          -          superuser
100000    3    tcp       0.0.0.0.0.111          -          superuser
100000    2    tcp       0.0.0.0.0.111          -          superuser
100000    4    udp       0.0.0.0.0.111          -          superuser
100000    3    udp       0.0.0.0.0.111          -          superuser
100000    2    udp       0.0.0.0.0.111          -          superuser
100000    4    local     /var/run/rpcbind.sock  -          superuser
100000    3    local     /var/run/rpcbind.sock  -          superuser
100005    1    udp       0.0.0.0.3.124          -          superuser
100005    1    tcp       0.0.0.0.3.124          -          superuser
100005    1    udp6      ::.3.124               -          superuser
100005    1    tcp6      ::.3.124               -          superuser
100005    2    udp       0.0.0.0.3.124          -          superuser
100005    2    tcp       0.0.0.0.3.124          -          superuser
100005    2    udp6      ::.3.124               -          superuser
100005    2    tcp6      ::.3.124               -          superuser
100005    3    udp       0.0.0.0.3.124          -          superuser
100005    3    tcp       0.0.0.0.3.124          -          superuser
100005    3    udp6      ::.3.124               -          superuser
100005    3    tcp6      ::.3.124               -          superuser
100003    2    tcp       0.0.0.0.8.1            -          superuser
100003    3    tcp       0.0.0.0.8.1            -          superuser
100003    4    tcp       0.0.0.0.8.1            -          superuser
100003    2    udp       0.0.0.0.8.1            -          superuser
100003    3    udp       0.0.0.0.8.1            -          superuser
100003    2    tcp6      ::.8.1                 -          superuser
100003    3    tcp6      ::.8.1                 -          superuser
100003    4    tcp6      ::.8.1                 -          superuser
100003    2    udp6      ::.8.1                 -          superuser
100003    3    udp6      ::.8.1                 -          superuser
100021    1    udp       0.0.0.0.223.27         -          superuser
100021    3    udp       0.0.0.0.223.27         -          superuser
100021    4    udp       0.0.0.0.223.27         -          superuser
100021    1    tcp       0.0.0.0.228.29         -          superuser
100021    3    tcp       0.0.0.0.228.29         -          superuser
100021    4    tcp       0.0.0.0.228.29         -          superuser
100021    1    udp6      ::.173.76              -          superuser
100021    3    udp6      ::.173.76              -          superuser
100021    4    udp6      ::.173.76              -          superuser
100021    1    tcp6      ::.135.197             -          superuser
100021    3    tcp6      ::.135.197             -          superuser
100021    4    tcp6      ::.135.197             -          superuser
100024    1    udp       0.0.0.0.201.171        -          superuser
100024    1    tcp       0.0.0.0.206.188        -          superuser
100024    1    udp6      ::.133.43              -          superuser
100024    1    tcp6      ::.130.4               -          superuser
admin@nas:/$ cat /etc/nfs/syno_nfs_conf 
udp_read_size=32768
udp_write_size=32768
nfsv4_enable=yes
nfs_unix_pri_enable=1
statd_port=0
nlm_port=0
nfs_custom_port_enable=no
admin@nas:/$ showmount -e
Export list for sigma:
/volume1/nfs    *
/volume1/files  *
/volume1/backup 10.42.0.0/24

BTW, how do I pass the working mount options to AutoFS? I found there is

# /etc/autofs/autofs.conf
# mount_nfs_default_protocol=3
# append_options = yes

but couldn'f find any info/doc on passing argument.

Last edited by dot1910 (2021-11-27 16:19:28)

Offline

#8 2021-11-27 14:12:11

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [solved] NFS client not working

Which one(s)?
"nolock", "proto=udp,mountproto=udp" or "sec=none"?
If UDP works by tcp doesn't it might be due to https://wiki.archlinux.org/title/Networ … tification or https://wiki.archlinux.org/title/Networ … eue_length

"sec" should™ affect 3 & 4 and nolock would suggest that 3 & 4 are plagued by different problems…

Offline

#9 2021-11-27 16:22:02

dot1910
Member
Registered: 2021-11-24
Posts: 6

Re: [solved] NFS client not working

proto=udp,mountproto=udp

Also updated on the previous post. DSM config was set to "sec=sys". So I didn't test this one out.

I'll test

- udp on NFSv4 server
- read two links you provided (Explicit Congestion Notification & MUT/txqueuelen)

Thanks a lot.

UPDATE:

- disabling "ECN" (from 2 to 0) does not work
- MTU: 1500, txqueuelen: 1000 on all clients and server

Guest I have to find how to pass options to AutoFS nfs auto mount.

Last edited by dot1910 (2021-11-27 16:49:35)

Offline

#10 2021-11-28 02:31:43

dot1910
Member
Registered: 2021-11-24
Posts: 6

Re: [solved] NFS client not working

It turns out my switch has 'DoS attack prevention' settings that is blocking the NFS SYNC. This is the post I get the hint.

I turned off the feature and now without extra options I can mount v3 and v4 now.

Offline

#11 2021-11-28 07:21:07

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [solved] NFS client not working

If the NAS allows unprivileged ports, you could try whether "noresvport" allows you to keep the "Actual Usage Prevention" enabled.

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Offline

Board footer

Powered by FluxBB