You are not logged in.

Pages: 1

Topic closed

#1 2013-11-20 05:54:04

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

[SOLVED] Samba share slow reads

I've been recently having a problem with very slow read speeds using samba (to serve shares).  I noticed this issue because I recently changed around my file server's drive configuration.  I have 2 different arch boxes with different disk configurations (one is just a general purpose headless linux box [isis] and the other is my file server [osiris]) and the same issue exists on shares served up from either box.

Windows (physical machines and a even a virtual) and Linux clients show the same behavior when reading from a samba share.  All the machines are on a GigE switch.  NFS, sftp and local transfers are fast.

Client write speeds are line rate (100+MB/s), but the read speeds are awful (<10MB/s).  Interestingly though, if I bounce the smbd and nmbd daemons and immediately try to transfer a file again, sometimes I can get 100+ MB/s.  The next file transfer though is back to <10MB/s.  During the slow transfer atop shows that nothing is maxed out (cpu, disk, network).

The issue has to be on the server side because a transfer reading from a windows share is fine, but a transfer from a linux machine running samba is not.

Things I have tried with no avail:
- Reseating ethernet cables on both ends/power cycling switch
- Ensuring all machines are 1000Mb/s Full Duplex with ethtool/windows
- Adding some samba tuning parameters (other than aio size, though I have never needed anything additional in the past)
   - read/write raw
   - sendfile
   - a few socket options
- Rolling back samba a few versions
- Different machines hosting shares

Specs:
[isis]
cpu: Intel Pentium G2020
memory: 8GB
disk: 2x 160GB 7200RPM drives in Hardware RAID-1 (Adaptec 6405E)
NIC: onboard realtek

[osiris]
cpu: Intel G620
memory: 16GB
disk: 4x 3TB 7200RPM Drives in 1 RAID-Z pool
NIC: onboard realtek

Here is my smb.conf from the file server.  This read performance used to be fine with this exact config file.  I have not changed it in quite some time.

#======================= Global Settings =====================================
[global]

workgroup = WORKGROUP

server string = osiris

printcap name = /dev/null
load printers = no

max log size = 1000

domain master = no
preferred master = no

# Name Resolution
name resolve order = wins lmhosts bcast
wins support = no
wins server = 192.168.0.101
dns proxy = no

# Security
security = user
guest account = guest
map to guest = Bad User

# Default Permissions
force user = guest
force group = root
create mask = 0775
force create mode = 0775
directory mask = 0775
force directory mode = 0775

#============================ Share Definitions ==============================

# NOTE: If you have a BSD-style print system there is no need to
# specifically define each individual printer
[printers]
   comment = All Printers
   path = /var/spool/samba
   browseable = no
# Set public = yes to allow user 'guest account' to print
   guest ok = no
   writable = no
   printable = yes

# Custom Parameters
aio read size = 1
aio write size = 1

# ZFS Shares
[media]
   path = /mnt/data/media
   public = yes
   only guest =  yes
   writable = yes
   printable = no
   vfs object = recycle
   recycle:repository = .recycle
   recycle:keeptree = yes
   recycle:directory_mode=777
   recycle:exlude_dir=.recycle

[backup]
   path = /mnt/data/backup
   public = yes
   only guest =  yes
   writable = yes
   printable = no

[documents]
   path = /mnt/data/documents
   public = yes
   only guest =  yes
   writable = yes
   printable = no

Here is my simpler smb.conf on the other server that exibits the same (slow) behavior on reads only:

[global]
   workgroup = WORKGROUP
   server string = isis
   printcap name = /dev/null
   load printers = no
guest account = guest
   log file = /var/log/samba/%m.log
   max log size = 1000
   security = user
   local master = yes
   domain master = yes
   preferred master = yes
 name resolve order = wins lmhosts bcast
   wins support = yes
   dns proxy = yes
[printers]
   comment = All Printers
   path = /var/spool/samba
   browseable = no
   guest ok = no
   writable = no
   printable = yes
[samba]
   path = /srv/samba/samba
   public = yes
   only guest = yes
   writable = yes
   printable = no
   create mask = 0665
   directory mask = 0775
   force group = users
[ftp_anonymous]
   path = /srv/samba/ftp_anonymous
   public = yes
   only guest =  yes
   writable = yes
   printable = no
   create mask = 0664
   directory mask = 0775
   force user = server
   force group = ftp

I'm not seeing any errors in the samba logs or on the hosts.  Does anyone have any idea what may be causing this/know a fix?

Thank you in advance,

Matt

Last edited by mattbarszcz (2013-11-21 01:23:25)

Offline

#2 2013-11-20 14:40:57

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: [SOLVED] Samba share slow reads

Hi,

which logs did you look at? I recently had a similar issue with some errors about locking in "clients pecific logs", aka /var/log/samba/$client.log like this: 

vm.log.3:[2013/06/01 21:47:01.524635,  0] ../source3/smbd/oplock.c:333(oplock_timeout_handler)
vm.log.3:  Oplock break failed for file file.end -- replying anyway

I don't know if you looked in these logs already, but if you didn't and there are similar errors, the option 

kernel oplocks = yes

solves this.

i'm sorry for my poor english wirting skills…

Offline

#3 2013-11-20 14:51:02

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

Re: [SOLVED] Samba share slow reads

andy123 wrote:

Hi,

which logs did you look at? I recently had a similar issue with some errors about locking in "clients pecific logs", aka /var/log/samba/$client.log like this: 

vm.log.3:[2013/06/01 21:47:01.524635,  0] ../source3/smbd/oplock.c:333(oplock_timeout_handler)
vm.log.3:  Oplock break failed for file file.end -- replying anyway

I don't know if you looked in these logs already, but if you didn't and there are similar errors, the option 

kernel oplocks = yes

solves this.

I checked through all the logs in the /var/log/samba directory and I didn't see any oplock timeouts, only a few init_oplocks.  I tried adding the option anyway, but it unfortunately it didn't have any affect.

I also tried this out in a VM this morning on the lts kernel 3.10.19-1-lts with the same results.

Thanks,
Matt

Last edited by mattbarszcz (2013-11-20 20:37:20)

Offline

#4 2013-11-20 21:43:24

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

Re: [SOLVED] Samba share slow reads

Some more information:

I setup the scenario as several VMs, 2 servers and an arch linux client.

The servers are:
[ssbxlinux02] - Server 1 - Arch CLI Only
- Arch: x86_64
- Kernel: 3.10.19-1-lts
- Samba 4.1.1

[ssbxlinux03] - Server 2 - CentOS 6.4
- Arch: x86_64
- Kernel 2.6.32-358.23.2.el6.x86_64
- Samba 3.6.9-151.el6_4.1

[ssbxlinux04] - Client - Arch with KDE

I used identical smb.conf files on each server:

[mbarszcz@ssbxlinux02 ~]$ cat /etc/samba/smb.conf
#======================= Global Settings =====================================
[global]

#Logging
log file = /var/log/samba/%m.log
max log size = 100
log level = 3

workgroup = WORKGROUP

server string = ssbxlinux02

printcap name = /dev/null
load printers = no

max log size = 1000

domain master = no
preferred master = no
kernel oplocks = yes

# Name Resolution
name resolve order = wins lmhosts bcast
wins support = no
dns proxy = no

# Security
security = user
guest account = guest
map to guest = Bad User

# Default Permissions
force user = guest
force group = root
create mask = 0775
force create mode = 0775
directory mask = 0775
force directory mode = 0775

#============================ Share Definitions ==============================

# NOTE: If you have a BSD-style print system there is no need to
# specifically define each individual printer
[printers]
   comment = All Printers
   path = /var/spool/samba
   browseable = no
# Set public = yes to allow user 'guest account' to print
   guest ok = no
   writable = no
   printable = yes

# Custom Parameters
#aio read size = 1
#aio write size = 1

# Shares
[test]
   path = /srv/samba/test
   public = yes
   only guest =  yes
   writable = yes
   printable = no

I then mounted these shares up on ssbxlinux04 and read a 5GB file filled with zero data.

[root@ssbxlinux04 ~]# time cp /mnt/samba_arch/zero.bin /dev/null
real 0m56.862s
user 0m0.090s
sys 0m6.840s

[root@ssbxlinux04 ~]# time cp /mnt/samba_centos/zero.bin /dev/null
real 38.224s
user 0m0.090s
sys 0m6.920s

This works out to a 84.9MB/s read from the arch share and 130MB/s read from the centos share.

Granted, these are VMs on the same physical hardware (and the same vSwitch), and I am moving zeros, but there is still nearly a 50% speed difference that seems to be even more significant on physical hardware.

Since my real file server has a Realtek nic (and the VMs have E1000), perhaps in addition to whatever performance difference came from the different samba versions/kernel versions, perhaps there was some change to the realtek network driver?  Seems strange though that NFS doesn't exhibit the same behavior.

Last edited by mattbarszcz (2013-11-20 21:59:45)

Offline

#5 2013-11-21 00:38:59

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

Re: [SOLVED] Samba share slow reads

I may be getting closer to a solution.

My first step was to try an intel based server NIC, no immediate change, though the behavior described below does not seem to occur on the onboard nic :-\

After transfering some much larger files, I noticed an interesting pattern.  The first 1GB (or so) of a file (or several large files) transfers slowly, and then kicks into gear and ramps up to the full 100MB/s for the duration of the copy job.

I experimented by disabling tcp_window_scaling with sysctl -w "net.ipv4.tcp_window_scaling=0" thinking that it may be related, but the ramp up took equally long with it on and off.

Below are snapshots of bmon graphs on the server where you can the speeds ramp after about 45 seconds:

RX    MiB                        [Bytes]                                (sel)
     2.61 ....********................................................
     2.18 ....*********...............................................
     1.74 ....*********...............................................
     1.31 ...**********...............................................
     0.87 ...*******************************************************..
     0.44 :::********************************************************* [-0.01%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s
TX    MiB
   111.32 ....********................................................
    92.76 ....*********...............................................
    74.21 ....*********...............................................
    55.66 ...**********...............................................
    37.11 ...*******************************************************..
    18.55 :::********************************************************* [-0.01%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s
RX    MiB                        [Bytes]                                (sel)
     2.67 .*****************..........................................
     2.23 ******************..........................................
     1.78 *******************.........................................
     1.34 *******************.........................................
     0.89 *******************.........................................
     0.45 ************************************************************ [-0.01%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s
TX    MiB
   114.51 .*****************..........................................
    95.43 *******************.........................................
    76.34 *******************.........................................
    57.26 *******************.........................................
    38.17 *******************.........................................
    19.09 ************************************************************ [-0.01%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s

Does this sound like the behavior of any particular tuning parameter samba or otherwise?

Last edited by mattbarszcz (2013-11-21 01:10:08)

Offline

#6 2013-11-21 01:22:18

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

Re: [SOLVED] Samba share slow reads

Solution:  It was the realtek drivers, not sure of the strange behavior on the Intel nic though.  At least it led me in the right direction.

lspci -v
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)
        Subsystem: Biostar Microtech Int'l Corp Device 230a
        Flags: bus master, fast devsel, latency 0, IRQ 43
        I/O ports at e000 [size=256]
        Memory at d0004000 (64-bit, prefetchable) [size=4K]
        Memory at d0000000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 2d-01-00-00-68-4c-e0-00
        Kernel driver in use: r8169
        Kernel modules: r8169, r8168

Install the r8168 drivers from community, blacklist the 8169 drivers as suggested by the install:

pacman -S r8168
echo "blacklist r8169" > /etc/modprobe.d/r8169_blacklist.conf

Reboot

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)
        Subsystem: Biostar Microtech Int'l Corp Device 230a
        Flags: bus master, fast devsel, latency 0, IRQ 42
        I/O ports at e000 [size=256]
        Memory at d0004000 (64-bit, prefetchable) [size=4K]
        Memory at d0000000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 00-00-00-00-68-4c-e0-00
        Kernel driver in use: r8168
        Kernel modules: r8169, r8168

# lsmod | grep r816
r8168                 255298  0

Performance = smile

RX    MiB                        [Bytes]                                (sel)
     2.50 ************************....................................
     2.08 ************************....................................
     1.66 ************************....................................
     1.25 ************************....................................
     0.83 ************************....................................
     0.42 *************************::::::............................. [-0.01%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s
TX    MiB
   112.48 ************************....................................
    93.74 ************************....................................
    74.99 ************************....................................
    56.24 ************************....................................
    37.49 ************************....................................
    18.75 *************************::::::............................. [-0.01%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s

Offline

#7 2013-11-28 06:54:36

solarwind
Member
From: Toronto
Registered: 2008-03-18
Posts: 546

Re: [SOLVED] Samba share slow reads

Hi, thanks so much for posting! I have the EXACT same card and same issue. I installed the r8168 drivers and blacklisted the other ones and it helps quite a bit, but there's still a short delay before the speed ramps up. I'll post if I figure anything else out.

Offline

#8 2013-11-28 18:39:25

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

Re: [SOLVED] Samba share slow reads

I had another machine that compiling and installing the r8168-all package from the AUR worked better than the one from community.  You might want to give that a shot.

Last edited by mattbarszcz (2013-11-28 18:39:44)

Offline

#9 2013-11-28 18:48:50

solarwind
Member
From: Toronto
Registered: 2008-03-18
Posts: 546

Re: [SOLVED] Samba share slow reads

mattbarszcz wrote:

I had another machine that compiling and installing the r8168-all package from the AUR worked better than the one from community.  You might want to give that a shot.

Just tried that, I didn't notice any difference.

Offline

#10 2013-12-14 08:10:10

akey
Member
Registered: 2012-02-09
Posts: 2

Re: [SOLVED] Samba share slow reads

Thanks for sharing!  I was doubting my pre-installed cables and about to invest into 802.11ac network until I saw your post.  Really solved my problem.

Offline

#11 2013-12-15 00:53:48

mattbarszcz
Member
Registered: 2012-06-25
Posts: 35

Re: [SOLVED] Samba share slow reads

akey wrote:

Thanks for sharing!  I was doubting my pre-installed cables and about to invest into 802.11ac network until I saw your post.  Really solved my problem.

I'm glad it was able to help you.  Out of curiosity, what revision is your controller (according to lspci)?  I have another machine that has a rev 09 controller, and the built in r8169 driver works fine.

Offline

#12 2015-10-27 19:18:22

liviucmg
Member
Registered: 2015-02-11
Posts: 3

Re: [SOLVED] Samba share slow reads

Thank you mattbarszcz, I had the exact problem, same rev 06. smile

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
        Subsystem: ASRock Incorporation Motherboard (one of many)
        Flags: bus master, fast devsel, latency 0, IRQ 32
        I/O ports at d000 [size=256]
        Memory at d0004000 (64-bit, prefetchable) [size=4K]
        Memory at d0000000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169
        Kernel modules: r8169

Last edited by liviucmg (2015-10-27 19:18:54)

Offline

#13 2015-10-27 20:10:40

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 20,202

Re: [SOLVED] Samba share slow reads

Closing

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

Pages: 1

Topic closed

Board footer

Powered by FluxBB