Mysterious NFS and systemd issue

MikeDacre · 2014-11-17 20:24:27

I don't know what to do with this problem

I have a cluster that runs arch linux. 20 compute nodes, a fast login node, and a head node that runs scientific linux 6.2 and has two large RAID drives attached (it also provides DNS, DHCP, and torque servers). The cluster is connected by a DDR infiniband network, including a direct infiniband connection between the login node and the head node. That infiniband has IPoIB set up, and an NFSv3 filesystem running over that. For years it just worked, with read/write speeds over 600MB/s, but in the last week or two it has started having lots of trouble. It is just really slow, sometimes the whole login node is slow for no good reason, but mostly that fast network drive is crushingly slow, with no heavy traffic. The problem is the same regardless of which network device I use, ethernet or infiniband. Plus, the login node itself has frozen and crashed several times in the last week. Major stability issues are just appearing out of nowhere.

Here is the best part though: I can see *nothing* useful in journalctl, or in any old-fashioned syslogs, on any of the nodes. Nothing. The only thing I see that is out of place is the following message repeating about once every three minutes:

Nov 17 08:09:47 fraser-server systemd[6962]: Starting Exit the Session...
Nov 17 08:09:47 fraser-server systemd[6962]: Reached target Shutdown.
Nov 17 08:09:47 fraser-server systemd[6962]: Starting Shutdown.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopped target Sockets.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopping Sockets.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopped target Timers.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopping Timers.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopped target Paths.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopping Paths.
Nov 17 08:09:47 fraser-server systemd[6962]: Stopped target Basic System.
Nov 17 08:09:46 fraser-server systemd[6962]: Stopping Basic System.
Nov 17 08:09:46 fraser-server systemd[6962]: Stopped target Default.

I get the distinct impression that that is completely unrelated though.

The output of nfsstat -v for both servers are here:

https://gist.github.com/MikeDacre/2d1b3e9c4bd17c96c97f

I am pretty sure this isn't the right place to post this problem, probably I should be emailing an infiniband or nfs forum, but with the complete lack of log information or any sign of hardware trouble, I really don't even know where to begin.

Anyone have any idea on where else I should be looking? Any good debug options I should try?

Arch Linux

#1 2014-11-17 20:24:27

Mysterious NFS and systemd issue

Board footer