Speeding up rsync (Gigabit NIC)

marcob · 2007-11-28 18:33:22

I'm replacing a fileserver running 2.6.20-ARCH and need to transfer about 2.9 Terabytes from the old to the new (also running 2.6.20-ARCH.) Both servers are from SuperMicro, the first has an 8 drive (each drive = SATA, 500 GB) hardware RAID5 array running from a 3ware 9550SX raid controller, and the second has an 8 drive (each drive = SATA, 1 TB) hardware RAID5 array running from a 3ware 9650SE raid controller. Each raid array runs a single partition formatted with XFS and each server runs Arch from a separate SATA drive.

I'm attempting to rsync the contents of the first to the second, but it's taking a lot longer than I would like. Both servers are connected with cat5e cables to a HP Procurve Gigabit switch that is about thirty feet away. The best speed rsync is showing is just under 8 Megabytes per second (~60 megabits per second.) There are big files being transferred, a lot of DVD images and CD images. Each server has an onboard dual port ethernet controller, both using the e1000 module. The first is an Intel 82546EB and the second is an Intel 80003ES2LAN.

I installed nttcp and it indicates that my gigabit connection is good, so I'm guessing it's either the performance of the array or the filesystem.

Does anybody have tips to improve rsync speed in this kind of situation? I've add the -W option to my rsync options, (copies files whole without rsync algorithm) and that helped out a little. My other rsync options are -avz, and I'm going to see what happens if I drop the 'z' option, which compresses the data during the transfer. Seems to me that would slow it down, but I'll see.

marcob · 2007-11-28 18:46:19

OK, turning off compression seemed to have a major effect. I'm now running at right about 15 Megabytes per second, which is double what I was running before, and I'm guessing if I didn't have the transfer capped at 16 MB/s, I would be running faster. Load average is way down as well. Durrrr, live and learn I guess.

cactus · 2007-11-28 21:28:13

yeah. compression adds lots of overhead 'on system'.
Some public rsync servers force compression off, in fact. I am pretty sure some of the gentoo rsync mirrors do that, for instance.

They have the bandwidth, and compression just uses too much cpu for their needs I guess.

marcob · 2007-11-28 23:31:53

I had to wait until they closed for the day to really test it, cooking along at 95 MB/s, dang...

Arch Linux

#1 2007-11-28 18:33:22

Speeding up rsync (Gigabit NIC)

#2 2007-11-28 18:46:19

Re: Speeding up rsync (Gigabit NIC)

#3 2007-11-28 21:28:13

Re: Speeding up rsync (Gigabit NIC)

#4 2007-11-28 23:31:53

Re: Speeding up rsync (Gigabit NIC)

Board footer