Hard Drive Cloning

Llama · 2015-07-22 02:37:20

Hi,

I've just bought a new hard drive, 3 Tb. I want my old disk cloned as is, nothing more. I'm trying the simplest approach from the Wiki:

$ sudo dd if=/dev/sda of=/dev/sdb bs=512 conv=noerror,sync

It's been running for 11 hours, and no end in sight. What am I doing wrong?

Trilby · 2015-07-22 02:51:56

What are you doing wrong? Using DD on a 3 TB drive!

How are the drives connected?

Are you really using anywhere near the 3 TB?

progandy · 2015-07-22 03:06:11

How are the drives connected and what read/write speed do your two drives provide? If everything works out to 100MB/s average then you should be able to clone 3TB in about 9 hours.
By the way, I think dd will work a bit better with a larger block size like bs=16M

If you want to get a status report, run a "killall -USR1 dd" in a second terminal, then view the output in the one with dd.

Edit: Please be sure /dev/sda* and /dev/sdb* is not mounted or your new copy will probably be corrupt. If arch is installed on /dev/sda, then I suggest you boot the install cd and use that to clone the disks.

Last edited by progandy (2015-07-22 03:12:51)

Llama · 2015-07-22 03:13:24

Trilby wrote:

What are you doing wrong? Using DD on a 3 TB drive!

OK. What else should I use?

How are the drives connected?

sda is my regular SATA drive, target sdb on USB.

Are you really using anywhere near the 3 TB?

Actual volume is less that 1.5 Tb.

frostschutz · 2015-07-22 07:24:59

bs=512 will copy using 4096 syscalls (2048x read(), 2048x write()) per MiB, too much for the CPU to handle. That would be ~500k syscalls per second for 100MiB/ish copy speeds - and syscalls, particularly I/O syscalls, are slow.

bs=16M on the other hand is too large, particularly with conv=noerror, it will leave you with a 16M hole for a single bad sector. Basically with noerror you want to pick the smallest blocksize that is still performant.

For noerror, bs=4k is better, it will already give you normal copy speeds compared to bs=512 and 4k is the blocksize used by filesystems anyway.

Optimal speeds are around bs=64k bs=128k bs=256k

bs=1M is the human readable option, it only gets slower after that. If the blocksize is too large, the disks will no longer work in parallel, instead one disk will idle while the other is writing or reading. Thus the speed cuts in half.

With one exception: If you're copying from/to the very same disks (e.g. moving partitions) a very large blocksize (bs=128M - bs=1G) lets the disk work linear instead of doing tons of seeks.

If you started off a slow dd you can cancel it and resume it, if you know how much data was already copied. For example if you're sure that you already copied a certain amount of data you could do something like this:

# dd if=/dev/source of=/dev/target bs=1M seek=42000 skip=42000

that would resume at the 42000MiB position. You should pick this value a bit smaller then the amount of data you're sure you've already copied. Better to copy a small region again, than to misjudge and leave a small region uncopied.

Last edited by frostschutz (2015-07-22 07:46:45)

Trilby · 2015-07-22 10:51:10

sdb being on a usb is the most impotant point here. There is a good range in the estimates depending on your exact hardware, but here is an example giving usb-connected-harddrive write speeds around 30MB/s. With that the estimated transfer time would be would be 12.1 days. This is making a lot of assumptions: first that this 3MB/s number if good - it might not be, you might get much better write speeds. But on the other side, this 12 day estimate is also assumming that whatever transfer speed you get can be maintained for a very extended period of time. Also this is calculating on the USB IO only. If we use the information about the number of system calls above, this 12 days would likely turn into something on the order of months.

You ask about other options - this is why I asked whether you were actually using the full 3TB. If that drive is filled to the brim with data you want transferred, then there likely are not better options, but you should definitely get the target drive connected to a sata connector or something other than usb. If you are not using the full 3TB, just partition the new drive in a similar way as the first, use rsync, and then adjust the fstab and bootloader on the target drive.

frostschutz · 2015-07-22 10:54:44

USB is not a problem... if it's USB3.

USB2 is a bit slow but still not as slow as dd with bs=512.

> 12.1 days

1.21?

If the dd is still running you can get a progress indication using killall -SIGUSR1 dd

Last edited by frostschutz (2015-07-22 10:56:06)

Trilby · 2015-07-22 11:10:27

No, defintely 12.1. I don't know what the actual transfer speeds are, but just using the 3MB/s value from the link I provided we get 3 x 1024 x 1024 MB / 3 = 1048576 seconds. 1048576 / 3600 = 291.2711 hours. 291.2711 / 24 = 12.1363 days.

frostschutz · 2015-07-22 11:12:46

3MB/s is silly. USB 2 is ~30MB/s.

Trilby · 2015-07-22 11:21:37

3 may be too low, but remember, we're talking about *write* speed over the usb. I've found other estimates that put that around 10-15MB/s under the best of conditions. So maybe 3-4 days.

frostschutz · 2015-07-22 11:31:05

...

If you have a USB2 hard disk that gets only 15MB/s writing, I'd return it. That's not normal.

USB sticks are a different matter.

Last edited by frostschutz (2015-07-22 11:32:05)

drcouzelis · 2015-07-22 15:10:04

For reference, cloning my 640 GB drive to an identical drive via a USB (2?) enclosure using "dd" takes me 5h30m.

I'm considering using CloneZilla in the future, since it might have some "tricks" to speed things up.

ewaller · 2015-07-22 17:03:12

drcouzelis:

That is 27 MBytes/sec. It is also about 218 MBits/Sec. That is a pretty good chunk of the 280 MBit/sec rate that can be generally achieved on USB 2.0. No doubt, the rest of the bandwidth is being used for overhead.

If data on the drive is sparse, then it is possible to speed things up by not copying the empty chunks. If the drive is full, the only thing that would help is a bigger pipe, or lossless compression.

Edit: Added 2.0

Llama: Of course, you should plan on this taking 3/0.6 times longer. That ratio is 5x what drcouzelis's transfer takes (Assuming the clone is a USB 2.0 device)

Last edited by ewaller (2015-07-22 17:07:47)

andrekp · 2015-07-23 12:07:22

FWIW, I find it's usually easier to just to a copy, retaining all attributes, from partition to partition, as suggested in the Arch wiki. You probably don't care if the disks are EXACT copies of each other, just that all files get moved and retain attributes. In the end, cp is safer and more fool proof than dd, I think.

ewaller · 2015-07-23 13:49:03

andrekp wrote:

In the end, cp is safer and more fool proof than dd, I think.

And, IMHO, rysnc is even safer and more fool proof. And can be interrupted and restarted. And can show progress.

drcouzelis · 2015-07-23 14:53:55

ewaller wrote:

andrekp wrote:
In the end, cp is safer and more fool proof than dd, I think.
And, IMHO, rysnc is even safer and more fool proof. And can be interrupted and restarted. And can show progress.

Using "rsync" is a great idea, and got me really excited! ...but then I remembered why I use "dd": other operating systems. Using "dd" also makes an exact copy of my partition table, Haiku installation, OpenBSD installation, and whatever else I can't remember I installed.

Anyway, I agree, if it works in your situation, "rsync" is a great solution!

mwillems · 2015-07-23 16:26:07

Rsync offers a lot of advantages over cp, not least of which is checksumming each copied file to make sure it didn't change as it was being copied. If a file is changed during a cp of that file, you get a silently corrupted "backup", which rsync would have detected and prevented. But rsync offers no assurance that two different files were not copied in an inconsistent state (it does not snapshot the whole filesystem, it just copies at the file level). A full system rsync can often take significant time allowing for lots of file changes in the interim, so I advise running rsync a second time immediately after the first run as a precaution if you're running rsync on a read-write mounted filesystem.

For many usage scenarios it doesn't really matter if files are modified during an rsync run (a lot of data is atomic and most files don't care what other files are doing), but if you have files that are updated automatically in the background and rely on other files being in a consistent state (virtual machines and snapshots of said machines, data with metadata stored in a separate file, etc.) a second rsync will provide some assurance that your backup files will be in a (more or less) consistent state. I have definitely gotten borked rsync backups of running virtual machines/snapshots before, so I learned my lesson.

Roken · 2015-07-23 21:15:02

Honestly, how many more threads asking to copy drives must we see? Search, people, search. This has been covered to death.

Slithery · 2015-07-23 21:33:37

I'm surprised no-one has mentioned Clonezilla. It's available as an Arch package but if your source drive contains your root partition then you're better off using the Live CD...

http://clonezilla.org/

Last edited by Slithery (2015-07-23 21:33:46)

2ManyDogs · 2015-07-23 21:45:05

slithery wrote:

I'm surprised no-one has mentioned Clonezilla. It's available as an Arch package but if your source drive contains your root partition then you're better off using the Live CD...

drcouzelis wrote:

I'm considering using CloneZilla in the future, since it might have some "tricks" to speed things up.

Slithery · 2015-07-23 21:51:27

It's late, I've had a long day...

Llama · 2015-07-27 02:40:19

slithery wrote:

I'm surprised no-one has mentioned Clonezilla. It's available as an Arch package but if your source drive contains your root partition then you're better off using the Live CD...
http://clonezilla.org/

It's worth a try, then. Copying by cp won't work on the big user partition, unlike the root you're warning me against.

Arch Linux

#1 2015-07-22 02:37:20

Hard Drive Cloning

#2 2015-07-22 02:51:56

Re: Hard Drive Cloning

#3 2015-07-22 03:06:11

Re: Hard Drive Cloning

#4 2015-07-22 03:13:24

Re: Hard Drive Cloning

#5 2015-07-22 07:24:59

Re: Hard Drive Cloning

#6 2015-07-22 10:51:10

Re: Hard Drive Cloning

#7 2015-07-22 10:54:44

Re: Hard Drive Cloning

#8 2015-07-22 11:10:27

Re: Hard Drive Cloning

#9 2015-07-22 11:12:46

Re: Hard Drive Cloning

#10 2015-07-22 11:21:37

Re: Hard Drive Cloning

#11 2015-07-22 11:31:05

Re: Hard Drive Cloning

#12 2015-07-22 15:10:04

Re: Hard Drive Cloning

#13 2015-07-22 17:03:12

Re: Hard Drive Cloning

#14 2015-07-23 12:07:22

Re: Hard Drive Cloning

#15 2015-07-23 13:49:03

Re: Hard Drive Cloning

#16 2015-07-23 14:53:55

Re: Hard Drive Cloning

#17 2015-07-23 16:26:07

Re: Hard Drive Cloning

#18 2015-07-23 21:15:02

Re: Hard Drive Cloning

#19 2015-07-23 21:33:37

Re: Hard Drive Cloning

#20 2015-07-23 21:45:05

Re: Hard Drive Cloning

#21 2015-07-23 21:51:27

Re: Hard Drive Cloning

#22 2015-07-27 02:40:19

Re: Hard Drive Cloning

Board footer