Best Way To Move Data

Carlwill · 2009-09-23 17:01:42

I am getting ready to tear down a server and I need to pull a directory off the server which has massive data on it. The directory is /data on "server_old" I would like to move that data as fast as possible to "server_new" but am not sure the best way to do so. Should I scp the data over SSH? Do I do some kind of Rsync command with compression switch enabled so the data is compressed and then sent over ssh to the "server_new"?

Any advice is greatly appreciated!

[root@server_old ~]# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sdb1     ext3    396G  364G   12G  97% /data

PS - Both servers are hooked up to a 1000MB switch and have Gigabit NIC's installed.

zen3 · 2009-09-23 17:40:45

rsync ?

tkdfighter · 2009-09-23 17:40:52

No, definitly don't use SCP, SFTP or rsync, which also uses SSH. SSH encrypts everything, which is very slow even on fast processors and beneficial only when moving stuff over an untrusted network. It would probably be fastest to use NFS or FTP to copy the data to your new server.

cactus · 2009-09-23 18:11:26

forklift

Carlwill · 2009-09-23 18:32:22

OK - I see this thread is going nowhere...

ckristi · 2009-09-23 21:48:47

If it's possible, just put your new hard drive inside of your old server and rsync your data with the "-ax" options to the new partition.

rsync -ax /olddir /newdir

-a -> archive mode, that means "preserve owner, group, user rights, symlinks, etc."
-x -> do not sync other filesystems mounted inside the one your copying (e.g. if you have mounted a partition in the /olddir directory, that partition is not copied)

fukawi2 · 2009-09-23 22:43:40

ckristi wrote:

If it's possible, just put your new hard drive inside of your old server and rsync your data with the "-ax" options to the new partition.
rsync -ax /olddir /newdir
-a -> archive mode, that means "preserve owner, group, user rights, symlinks, etc."
-x -> do not sync other filesystems mounted inside the one your copying (e.g. if you have mounted a partition in the /olddir directory, that partition is not copied)

That's probably your best option, or if you can't physical move the drives, just setup rsync on 'old_server' to listen for incoming connections. This doesn't use encryption so will be faster than tunnelling it over SSH.

Carlwill · 2009-09-24 12:34:48

So it looks like Rsync is the best bet with the exception that it uses encryption over SSH. I have never used NFS but I know that would require my configuring the NFS server daemon on the server under /etc/exports file and then having to download and install a NFS client on the new server so it can connect to the NFS server.

zukka · 2009-09-24 13:43:13

I've been told that ssh encryption is much faster using blowfish algorithm, so if you can't move the disk you could try with

rsync -e "ssh -c blowfish" -ax --numeric-ids /olddir user@newserver:/newdir

Last edited by zukka (2009-09-24 13:43:37)

Carlwill · 2009-09-24 13:46:46

Thanks! I will give the above listed a shot!

dmartins · 2009-09-24 14:10:19

NFS is not the best option because if the connection is broken you cannot be sure where you left off. rsync can be used over rsh which does no encryption (rsync -e rsh). Of course, you need rshd running, which can be started through xinetd or manually.

vacant · 2009-09-24 14:31:11

netcat is probably fastest. If you want to compress/encrypt as well it's up to you. I used the example in the link to copy over my localnet.

Last edited by vacant (2009-09-24 14:31:52)

lordmetroid · 2009-09-27 22:01:07

how about git?

fukawi2 · 2009-09-27 22:43:26

If the OP was trying to sync the 2 machines, and keep them in sync over time then git might be suitable, but for a one-time transfer from A or B, seems a bit of overkill given the other solutions available...

mikesd · 2009-09-27 23:59:27

I would use rsync for this. rsync only uses ssh if you tell it to. If both machines are on your local network you don't really need to encrypt the transfer as you would if sending the stream through the internet. Setup a rsync server on the old machine and use the rsync protocol instead of ssh.

Arch Linux

#1 2009-09-23 17:01:42

Best Way To Move Data

#2 2009-09-23 17:40:45

Re: Best Way To Move Data

#3 2009-09-23 17:40:52

Re: Best Way To Move Data

#4 2009-09-23 18:11:26

Re: Best Way To Move Data

#5 2009-09-23 18:32:22

Re: Best Way To Move Data

#6 2009-09-23 21:48:47

Re: Best Way To Move Data

#7 2009-09-23 22:43:40

Re: Best Way To Move Data

#8 2009-09-24 12:34:48

Re: Best Way To Move Data

#9 2009-09-24 13:43:13

Re: Best Way To Move Data

#10 2009-09-24 13:46:46

Re: Best Way To Move Data

#11 2009-09-24 14:10:19

Re: Best Way To Move Data

#12 2009-09-24 14:31:11

Re: Best Way To Move Data

#13 2009-09-27 22:01:07

Re: Best Way To Move Data

#14 2009-09-27 22:43:26

Re: Best Way To Move Data

#15 2009-09-27 23:59:27

Re: Best Way To Move Data

Board footer