You are not logged in.

#1 2010-02-06 11:42:13

seiichiro0185
Member
From: Leipzig/Germany
Registered: 2009-04-09
Posts: 226
Website

NFS write performance 6 times slower than read

Hi all,

I built my self a new homeserver and want to use NFS to export stuff to the clients. Problem is that I get a big difference in writing and reading from/to the share. Everything is connected by GBit Network, and raw network speed is fine.

Reading on the clients yields about 31MByte/s which is almost the native speed of the disks (which are luks-encrypted). But writing to the share gives only about 5.1MByte/s in the best case. Writing to the disks internally gives about 30MByte/s too. Also writing with unencrypted rsync from the client to the server gives about 25-30MByte/s, so it is definitely not a network or disk problem. So I wonder if there is anything that I could do to improve the Write-Performance of my NFS-shares. Here is my config which gives the best results so far:

Server-Side:

/etc/exports

/mnt/data               192.168.0.0/24(rw,async,no_subtree_check,crossmnt,fsid=0)
/mnt/udata              192.168.0.0/24(rw,async,no_subtree_check,crossmnt,fsid=1)

/etc/conf.d/nfs-server.conf

NFSD_OPTS=""
NFSD_COUNT="32"
PROCNFSD_MOUNTPOINT=""
PROCNFSD_MOUNTOPTS=""
MOUNTD_OPTS="--no-nfs-version 1 --no-nfs-version 2"
NEED_SVCGSSD=""
SVCGSSD_OPTS=""

Client-Side:

/etc/fstab

192.168.0.1:/mnt/data /mnt/NFS        nfs     rsize=32768,wsize=32768,intr,noatime    0 0

Additional Infos:
NFS to the unencrypted /mnt/udata gives about 20MByte/s reading and 10MByte/s writing.
Internal Speed of the discs is about 37-38MByte/s reading/writing for the encrypted one, and 44-45MByte/s for the unencrypted (notebook-hdd)

I Noticed that the load average on the server goes over 10 while the CPU stays at 10-20%

So if anyone has any idea what might go wrong here please let me know. If you need more information I will gladly provide it.

TIA
seiichiro0185

Last edited by seiichiro0185 (2010-02-06 13:05:23)


My System: Dell XPS 13 | i7-7560U | 16GB RAM | 512GB SSD | FHD Screen | Arch Linux
My Workstation/Server: Supermicro X11SSZ-F | Xeon E3-1245 v6 | 64GB RAM | 1TB SSD Raid 1 + 6TB HDD ZFS Raid Z1 | Proxmox VE
My Stuff at Github: github
My Homepage: Seiichiros HP

Offline

#2 2010-02-08 14:22:51

seiichiro0185
Member
From: Leipzig/Germany
Registered: 2009-04-09
Posts: 226
Website

Re: NFS write performance 6 times slower than read

*bump* doesn't anyone have an idea?


My System: Dell XPS 13 | i7-7560U | 16GB RAM | 512GB SSD | FHD Screen | Arch Linux
My Workstation/Server: Supermicro X11SSZ-F | Xeon E3-1245 v6 | 64GB RAM | 1TB SSD Raid 1 + 6TB HDD ZFS Raid Z1 | Proxmox VE
My Stuff at Github: github
My Homepage: Seiichiros HP

Offline

#3 2010-03-02 11:10:28

seiichiro0185
Member
From: Leipzig/Germany
Registered: 2009-04-09
Posts: 226
Website

Re: NFS write performance 6 times slower than read

I'll try to bump this up again..

maybe someone has an idea. Does anyone know how to see what exactly is causing the high load average? Any tipps on debugging this further would be really appreciated.


My System: Dell XPS 13 | i7-7560U | 16GB RAM | 512GB SSD | FHD Screen | Arch Linux
My Workstation/Server: Supermicro X11SSZ-F | Xeon E3-1245 v6 | 64GB RAM | 1TB SSD Raid 1 + 6TB HDD ZFS Raid Z1 | Proxmox VE
My Stuff at Github: github
My Homepage: Seiichiros HP

Offline

#4 2010-03-02 23:48:32

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: NFS write performance 6 times slower than read

Your rsize and wsize looks way too big. I just use defaults and it runs fine.
I don't know what your server is but I plucked this from BSD Magazine.

There is one point worth mentioning here, modern Linux usually uses wsize and rsize 8192 by default and that can cause problems with BSD servers as many support only wsize and rsize 1024. I suggest you add the option -o wsize=1024,rsize=1024 when you mount the share on your Linux machines.

You also might want to check here for some optimisations http://www.linuxselfhelp.com/howtos/NFS … WTO-4.html

A trick to increase NFS write performance is to disable synchronous writes on the server. The NFS specification states that NFS write requests shall not be considered finished before the data written is on a non-volatile medium (normally the disk). This restricts the write performance somewhat, asynchronous writes will speed NFS writes up. The Linux nfsd has never done synchronous writes since the Linux file system implementation does not lend itself to this, but on non-Linux servers you can increase the performance this way with this in your exports file:

Last edited by sand_man (2010-03-03 00:23:23)


neutral

Offline

#5 2010-03-03 07:21:56

seiichiro0185
Member
From: Leipzig/Germany
Registered: 2009-04-09
Posts: 226
Website

Re: NFS write performance 6 times slower than read

Thanks for your answer.

actually this is the rsize/wsize that gives the best performance. I did test all rsize/wsize values starting at 1024 up to 32768 and the given size gives the best performance in both directions.

The server is an arch machine like the client. I also tired setting sync/async and as you may see in my server config it is already set to async at the moment.


My System: Dell XPS 13 | i7-7560U | 16GB RAM | 512GB SSD | FHD Screen | Arch Linux
My Workstation/Server: Supermicro X11SSZ-F | Xeon E3-1245 v6 | 64GB RAM | 1TB SSD Raid 1 + 6TB HDD ZFS Raid Z1 | Proxmox VE
My Stuff at Github: github
My Homepage: Seiichiros HP

Offline

#6 2010-03-03 09:53:24

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: NFS write performance 6 times slower than read

seiichiro0185 wrote:

The server is an arch machine like the client. I also tired setting sync/async and as you may see in my server config it is already set to async at the moment.

So it is. smile
Sorry I'm out of ideas then.

Just as a test, do you get similar speeds when using rsync or scp?


neutral

Offline

#7 2010-03-03 10:08:46

seiichiro0185
Member
From: Leipzig/Germany
Registered: 2009-04-09
Posts: 226
Website

Re: NFS write performance 6 times slower than read

rsync (unencrypted) gives 25-30MB/s in both directions
scp (due to the Servers Atom having to handle 2 encryptions, for the link and the disks) gives between 15-20MB/s in both directions

Last edited by seiichiro0185 (2010-03-03 10:26:37)


My System: Dell XPS 13 | i7-7560U | 16GB RAM | 512GB SSD | FHD Screen | Arch Linux
My Workstation/Server: Supermicro X11SSZ-F | Xeon E3-1245 v6 | 64GB RAM | 1TB SSD Raid 1 + 6TB HDD ZFS Raid Z1 | Proxmox VE
My Stuff at Github: github
My Homepage: Seiichiros HP

Offline

Board footer

Powered by FluxBB