[SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

w201 · 2013-07-11 02:36:26

Hey everyone - so I purchased a cheap hard drive today to backup my system. I cloned the entire drive to the new drive, which has 80Gbs more space. I booted into the new drive and noticed that the additional 80 gigs is not allocated to any of the partitions. This is my first time cloning so I probably F'd something up, even though everything works fine. If you look below, sbd does show 149Gs. How can I regain that space and assign it to my home partition? Thanks!

***OLD***

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 75.8G  0 disk 
├─sda1   8:1    0   500M  0 part /boot/efi
├─sda2   8:2    0   500M  0 part /boot
├─sda3   8:3    0    20G  0 part /
├─sda4   8:4    0    20G  0 part /var
└─sda5   8:5    0  33.6G  0 part /home

***NEW***

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb      8:0    0 149.1G  0 disk 
├─sdb1   8:1    0   500M  0 part /boot/efi
├─sdb2   8:2    0   500M  0 part /boot
├─sdb3   8:3    0    20G  0 part /
├─sdb4   8:4    0    20G  0 part /var
└─sdb5   8:5    0  33.6G  0 part /home

Last edited by w201 (2013-07-12 09:14:50)

Lone_Wolf · 2013-07-11 12:01:28

What commands / program did you use to clone the drive ?

you should be able to use tools like parted / gparted to add the free space to the home partition on the new disk.
(boot from a live usb/cd to use those tools, if you're a bit lucky you will not even suffer data loss.)

an alternative might be to create an sdb6 partition and use lvm2 to combine sdb5 and sdb6 into 1 home.

Trilby · 2013-07-11 12:08:21

If you used a tool like dd, or most other cloning tools, then this is to be expected. Partitions have sizes, and when you copy partitions, you copy their size.

w201 · 2013-07-11 12:17:55

Hi Lone_Wolf,

I used...

dd if=/dev/sda of=/dev/sdb bs=4096 conv=notrunc,noerror

I'll try using parted as you suggested. For future reference, using the dd command, is it possible to clone a drive and instruct the computer to assign the additional space to a specific partition so that it's not lost? I didn't see any options like that on the man page.

Last edited by w201 (2013-07-11 12:18:27)

Trilby · 2013-07-11 12:21:57

No. dd does not create partitions, it doesn't see partitions, it doesn't know anything about partitions or even need the disk to be partitioned. dd just copies bits from one device to another.

If you use dd to copy a block device, you'll always have to adjust the partition size.

You can partition the target device appropriately first, then dd each partition instead of the whole device in one go. `dd if=/dev/sda1 of /dev/sdb1 && dd if=/dev/sda2 of=/dev/sdb2` instead of `dd if=/dev/sda of=/dev/sdb` as an example.

Or more likely you'll want to use something like rsync instead of dd. This has the added bonus of not wasting time copying unused space.

w201 · 2013-07-11 12:29:26

Trilby wrote:

No. dd does not create partitions, it doesn't see partitions, it doesn't know anything about partitions or even need the disk to be partitioned. dd just copies bits from one device to another.
If you use dd to copy a block device, you'll always have to adjust the partition size.
You can partition the target device appropriately first, then dd each partition instead of the whole device in one go. `dd if=/dev/sda1 of /dev/sdb1 && dd if=/dev/sda2 of=/dev/sdb2` instead of `dd if=/dev/sda of=/dev/sdb` as an example.
Or more likely you'll want to use something like rsync instead of dd. This has the added bonus of not wasting time copying unused space.

Trilby,

Thanks, that's exactly what I needed to know. I'm heading over to the rsync wiki as I haven't used it before, but I'll give that a try.

tomk · 2013-07-11 13:05:41

I've done this many times, and I always just use cp -a .

w201 · 2013-07-11 13:29:43

Based on what I've read so far on the rsync wiki, if you're backing up regularly, rsync has the advantage of backing up only the files that changed since the last backup...so the first backup takes a while, and subsequent backups are much faster.

**EDIT**

oops, I guess cp can do that too :-)

Last edited by w201 (2013-07-11 13:34:32)

Trilby · 2013-07-11 13:37:26

That's true, but it's relative. In either case rsync (or cp -a) will be faster than dd.

rysnc on the first backup, or cp -a, will copy every file on the device. dd will copy ever binary digit, whether it is in a file or not.

I'm not sure if rsync's incremental backup ability would be useful in your case. Generally this is done with a separate target path for each subsequent backup, so if a file hasn't changed, it just creates a symlink to the existing backup. If you are overwriting the same target there is nothing to link to, I don't know if rsyn can leave unchanged files alone (perhaps it can, but I've never done this).

Last edited by Trilby (2013-07-11 13:37:49)

w201 · 2013-07-11 14:52:22

Basically, I want a a second drive which I can back up to on a weekly or even monthly basis in case my drive fails, as my main drive is pushing 6 years. I know Western Digitals are rock solid, but I have had a few fail in the past. Regardless, I've already recovered the lost space, partitioned my drive and created the file system extensions, so it's ready for backup. I guess I have a few options, just not sure if I should go with cp or rsync at this point.

dag · 2013-07-11 15:26:22

if interupted rsync can resume from large files

Last edited by dag (2013-07-11 15:29:31)

R00KIE · 2013-07-11 18:28:48

I'd use 'rsync -aAHXP' otherwise some things are lost and will not work as expected.

w201 · 2013-07-11 20:54:36

I'm going with rsync. There's one thing that's not very clear to me and the wiki doesn't go into much detail. The wiki says for a full system backup use:

# rsync -aAXv /* /path/to/backup/folder --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found}

My drive is already partitioned. I'm confused on how to set the destination path because I have all 5 partitions mounted at /mnt/1, /mnt/2, etc.. That make any sense? ALso, on the first backup, do I leave out the exclude string? Sorry guys, I've had a long day and I'm probably over thinking this too much, but the destination path is confusing to me. How will rsync know exactly where to place every partition unless I tell it exactly where to go?

Last edited by w201 (2013-07-11 20:54:56)

Trilby · 2013-07-11 22:41:46

You have two choices for that. One, use several rsync commands, one for each partition; or two, mount the partitions under their appropriate names (eg root as /mnt, boot as /mnt/boot, home /mnt/home, etc) then use /mnt as the path/to/backup.

Those excluded directories should always be excluded - they are not 'on-disk' directories and are recreated at boot.

cfr · 2013-07-12 00:36:23

tomk wrote:

I've done this many times, and I always just use cp -a .

As mentioned above, this will lose some attributes and break some things. Not much but it will lose information which rsync can conserve (if you use the flags recommended in the wiki on full backup with rsync).

EDIT: https://bbs.archlinux.org/viewtopic.php … 3#p1299083 was the post I had in mind.

Last edited by cfr (2013-07-12 02:31:17)

tomk · 2013-07-12 01:31:31

cfr, nobody mentioned anything of the sort concerning cp -a. Please re-read the relevant posts, and do not spread FUD.

cfr · 2013-07-12 02:19:00

I interpreted the rsync comment that way. However, perhaps that was not intended. It is, nonetheless, true. I can't currently find the thread where this caused a failure. I think it was ping but I may be wrong about that.

$ getfacl /tmp/a/1
getfacl: Removing leading '/' from absolute path names
# file: tmp/a/1
# owner: <me>
# group: <me>
user::rw-
group::r--
group:between:rwx
mask::rwx
other::r--

$ rsync -aAXv /tmp/a/ /tmp/b

$ getfacl /tmp/b/1 
getfacl: Removing leading '/' from absolute path names
# file: tmp/b/1
# owner: <me>
# group: <me>
user::rw-
group::r--
group:between:rwx
mask::rwx
other::r--

$ cp -a /tmp/a /tmp/c

$ getfacl /tmp/c
getfacl: Removing leading '/' from absolute path names
# file: tmp/c
# owner: <me>
# group: <me>
user::rwx
group::r-x
other::r-x

As you can see, cp -a loses information. In most cases, this will not matter but in some cases, it will. I think I must be wrong about it being ping, though, because it doesn't seem to apply to ping any more. Nonetheless, it is certainly not FUD. Believe me, I come from OS X and this stuff really can matter. The fact that most stuff in Linux does not depend on it does not mean that nothing does even if I cannot now find the thread to prove it can actually break stuff. Moreover, on any system where ACLs are used, cp -a risks breaking permissions configured by the admin (though perhaps in this case the admin can be expected to know that cp -a will do this).

EDIT: The thread I was trying to find includes a comment towards the end where somebody mentions updating the wiki page on full system backup with rsync to include the necessary flags. The original incantation did not preserve all attributes either. The current recommendation preserves everything I've tested for but this is not, obviously, exhaustive. There used to be a standard test for OS X backup applications but I'm not aware of one for Linux. Maybe this is because Linux has not traditionally depended on this stuff whereas not only has OS X so depended, at one point no backup programme preserved even most of the information and many preserved almost none. Also, OS X relies much, much more extensively on extended attributes which are a minefield.

EDIT: That's wrong. So either cp -a does now preserve the information or I'm wrong about what it fails to preserve.

$ getfacl /tmp/c/1 
getfacl: Removing leading '/' from absolute path names
# file: tmp/c/1
# owner: <me>
# group: <me>
user::rw-
group::r--
group:between:rwx
mask::rwx
other::r--

Currently, I'm not sure which...

EDIT: It is capabilities I was thinking of and ping does depend on them but cp -a now seems to preserve capabilities just as rsync -aAX does. Sorry about that.

Last edited by cfr (2013-07-12 02:45:35)

w201 · 2013-07-12 09:08:18

Trilby wrote:

You have two choices for that. One, use several rsync commands, one for each partition; or two, mount the partitions under their appropriate names (eg root as /mnt, boot as /mnt/boot, home /mnt/home, etc) then use /mnt as the path/to/backup.
Those excluded directories should always be excluded - they are not 'on-disk' directories and are recreated at boot.

Trilby - That did the trick. This was a good learning experience! Thanks to everyone else who chimed in as well.

Arch Linux

#1 2013-07-11 02:36:26

[SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#2 2013-07-11 12:01:28

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#3 2013-07-11 12:08:21

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#4 2013-07-11 12:17:55

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#5 2013-07-11 12:21:57

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#6 2013-07-11 12:29:26

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#7 2013-07-11 13:05:41

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#8 2013-07-11 13:29:43

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#9 2013-07-11 13:37:26

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#10 2013-07-11 14:52:22

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#11 2013-07-11 15:26:22

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#12 2013-07-11 18:28:48

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#13 2013-07-11 20:54:36

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#14 2013-07-11 22:41:46

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#15 2013-07-12 00:36:23

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#16 2013-07-12 01:31:31

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#17 2013-07-12 02:19:00

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

#18 2013-07-12 09:08:18

Re: [SOLVED] Cloning From an 80G Drive to a 160G Drive - Lost Drive Space

Board footer