Arch Linux

PeterBell

Member

Registered: 2014-07-20

Posts: 39

[solved] nfs transfers hanging?

I have been using Arch on my desktop for several years and have never had problems accessing nfs shares on my Slackware server.

In the last couple of weeks, or so, accesses to nfs (v3) shares on the server have been hanging.  This is evident when doing file copies using Thunar, but first came to light using MKVToolNix GUI to manipulate mkv files, reading from one nfs share and writing back to another.  This is an operation I perform fairly regularly, so I know that the problem is new.

Performing exactly the same MKVToolNix operation on a Mint desktop, using the same shares on the same server works perfectly.  This leads me to believe that the problem lies with my Arch system - not with the server

On both desktop environments I use autofs to mount the shares.

I have no idea how to go about tracking this problem down and would be very grateful if someone was able to give me some advice.

Last edited by PeterBell (2016-02-28 04:01:59)

PeterBell

Member

Registered: 2014-07-20

Posts: 39

Re: [solved] nfs transfers hanging?

A little more information:

When MKVToolNix goes into its 'hang' state (this time it was 6% through processing the mkv), htop showed that one cpu was in 100% kernel.  The other three cpus were lightly loaded (3% or less).

Obviously, in this situation, I'm not dealing with 'lots of small reads or writes'.

One possibility, from the nfs troubleshooting topic in the wiki, is an intermittent client freeze.  The suggestion is to add the sync option on the mount, but I'm not clear where to add this in the autofs configuration.  I also don't understand why this problem has suddenly occured, about two weeks ago, when it's been working perfectly well for several years.  Apart from doing regular pacman -Syu updates, I haven't. knowingly, made any changes to the configuration/setup.

I guess that I may need to get into rpcdebug, but I fear that I'm rapidly getting out of my depth.

Last edited by PeterBell (2016-02-18 11:09:42)

x33a

Forum Fellow

Registered: 2009-08-15

Posts: 4,587

Re: [solved] nfs transfers hanging?

One possibility, from the nfs troubleshooting topic in the wiki, is an intermittent client freeze.  The suggestion is to add the sync option on the mount, but I'm not clear where to add this in the autofs configuration.

Perhaps you need to add it to your exports file. Please post your exports file.

PeterBell

Member

Registered: 2014-07-20

Posts: 39

Re: [solved] nfs transfers hanging?

The exports file on the server is auto-generated  (root filing system is in ram).  All the entries look like:

"/mnt/user/mysql" -async,no_subtree_check,fsid=108 *(sec=sys,rw,insecure,anongid=100,anonuid=99,all_squash)

I'm not sure that I have any control over the contents.

Again, I have not made any changes to the server, and it is still working perfectly well from another desktop client running a different distro.

Further Info:
I have been able to succesfully run MKVToolNix with source file on the server, destination on a local disk and with the source file on a local disk, destination on the server.  This would point to the problem occuring only when simultaneously reading and writing from/to the server.

The same seems to be true when using Thunar - I can copy a large file from desktop to server, or from server to desktop, but if I copy from one share to another the process hangs with one cpu stuck 100% in kernel.  Again, I can perform the same action on my Mint desktop without any problem

Last edited by PeterBell (2016-02-18 17:53:40)

Painless

Member

Registered: 2006-02-06

Posts: 233

Re: [solved] nfs transfers hanging?

What are your NFS mount options on the Arch client?  Have you compared this with the same on the Mint client?  Are you copying over wifi or ethernet?  For large files it's usually a good idea to tune rsize and wsize on the client.

PeterBell

Member

Registered: 2014-07-20

Posts: 39

Re: [solved] nfs transfers hanging?

Thanks for your response.

I am surprised to note that there are vast differences in the information returned by the 'mount' command on the two machines.

On both I am using a simple

/net	-hosts

line in the auto.master file.

However, while the mint m/c shows all the shares from my server in the form

tower:/mnt/user/Movies on /net/tower/mnt/user/Movies type nfs (rw,nosuid,nodev,intr,sloppy,addr=172.22.0.100)

whether they have been accessed or not,

the arch linux machine shows

-hosts on /net/tower/mnt/user/Movies type autofs (rw,relatime,fd=12,pgrp=564,timeout=300,minproto=5,maxproto=5,offset)

for each server share, whether accessed or not, while the share which has been accessed is shown as:

tower:/mnt/user/Movies on /net/tower/mnt/user/Movies type nfs (rw,nosuid,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.22.0.100,mountvers=3,mountport=37704,mountproto=udp,local_lock=none,addr=172.22.0.100)

So, I have a number of questions:
1. Why the great difference, even though autofs in configured identically (I believe) on the two machines?
2. What do the '-hosts' lines mean, particularly since they don't appear on the Mint machine?
3. What is controlling the difference in behaviour?
4. How can I change/control the behaviour?
5. Why has something changed on the Arch machine in the last couple of weeks?

Painless

Member

Registered: 2006-02-06

Posts: 233

Re: [solved] nfs transfers hanging?

1. Not sure, I have no experience with Mint.  Maybe something to do with "the Debian way".

2. "-hosts" refers to the automount map, in this case 'cd' to or 'ls' of /net/hostname will trigger autofs to automount the specific share.

3. Since your mount options appear to be identical on both clients, I'm not sure but see below.

4. You can pass NFS mount options in the currently unused third field in /etc/autofs/auto.master, see 'man auto.master'.  I don't think this will be required though from your visible mount options.  You could also carefully compare the NFS3 configuration on slack/arch/mint.  Do you have a firewall on the server or the Arch client?  If so you will need to specify statd_opts to use open ports for NFS3.  There may be other issues.

5. Not sure, autofs was rebuilt due to a key trust issue so effectively only the pkgrel number was changed, but make sure no config files were inadvertently changed.  Also check your network interface settings.  Check mkvtoolnix versions.  Check everything

PeterBell

Member

Registered: 2014-07-20

Posts: 39

Re: [solved] nfs transfers hanging?

Well, it would seem that the problem I was experiencing was the same as that covered in the nfs troubleshooting wiki page - I needed to add the 'sync' option to the mounts.

This is easily achieved with autofs (when you know how) by adding 'sync' in the third field on the appropriate line in auto.master (thank you very much, Painless):

/net	-hosts         sync

This appears to have fixed the hangs/delays - I have been able to run mkvtoolnix , reading from one share and writing to another and I have also been able to copy files from one share to another in Thunar.  I also get the impression that these nfs transfers are running faster than before.

So, my belief is that some recent change in Arch has caused the problem to afflict me, but that nfs transfers were never running as fast as they should have.

I suspect that this problem will also affect other distros, in time.  Arch is close to 'bleeding edge' while Mint lags a long way behind.  I will make the same change to auto.master on Mint and see whether transfers get faster.