how to create a bootable clone?

ataraxia · 2009-05-20 00:47:49

So, I'm returning after a while in Mac-land, and there's one thing I'm missing that the Mac got very right. There were several apps that would easily clone your system onto a backup disk and make it bootable. It was the best backup system I ever had. If my disk died, I just rebooted onto the other one, and when I got the replacement, I used basically the same process to do the restore.

How can I accomplish this in Arch? The "dd the disk" method isn't going to work out to well, as I need support for disks of different sizes, partition layouts, and interfaces. Rsync seems promising, but there's a fair bit of work to do after the copy since things like FS UUIDs will be different.

The way this works on Mac is basically like this:
1. rsync the source to the destination, excluding a few things that are useless, as recommended by Apple.
2. Make the clone bootable. This is easy on Mac as boot code is never written outside the root FS, and doesn't even need to be installed into the FS header - all that has to happen is that the inode number of the boot code is written to a specific spot in the HFS+ header.
3. The Mac's firmware (either PPC OpenBoot or X86 EFI) will scan all attached disks for bootable filesystems and show a list for you to pick, when interrupted with the right key.
4. The root FS is always the boot FS, which eliminates the need for configuring the root FS in the boot code on fstab. It just mounts whatever it booted from as root. This neatly eliminates the need to post-edit these files after a clone

As I see it, the difficulty of automating this process in Linux really has a lot to do with the lameness of the PC BIOS.

I don't want to roll my own cloning code. Backups are too important to trust it to the kind of dirty hack I'd come up with. What can I use?

darthaxul · 2009-05-20 02:06:06

Macs are for people that want things to work, even if a person doesnt have any computer knowledge.

If my disk died, I just rebooted onto the other one, and when I got the replacement, I used basically the same process to do the restore.
How can I accomplish this in Arch?

making a complete backup can be done with the cp command. then you just edit the fstab file for that copy.

jwcxz · 2009-05-20 02:46:13

dar is great for this task. That's how I transferred my Arch installation from my old hard drive to my new one on my laptop. My method was soemthing like:

1. Boot to live CD/USB OS.
2. Use dar to create a clone of Linux partition onto an external hard drive.
3. Replace old hard drive with new one.
4. undar partition onto new partition on the new drive
5. Install GRUB.

dar supports differential backups, so you can easily create a master file and then take subsequent backups every week or whatever. That's more or less how I backup my home directory.

Wikimig · 2009-05-20 06:35:01

making a complete backup can be done with the cp command. then you just edit the fstab file for that copy.

I did that and it works pretty well even for "porting" a Arch installation to another computer. I booted a liveCD so that nothing gets mounted in /mnt/ or anywhere else and then just did a tar.gz of the root!

Heller_Barde · 2009-05-20 06:41:00

ataraxia wrote:

I don't want to roll my own cloning code. Backups are too important to trust it to the kind of dirty hack I'd come up with. What can I use?

if you rsync your stuff to another hard disk with the same partition layout and stuff and also install GRUB there, you're all set, if my memory does not fail me. that should actually "just work"

the best way to do it if you ask me is this:

(I'll call the currently working disk "sda" and the backup disk "sdb")

sdb needs to be empty for now. sda needs to be all set and bootable and working for now.

1. copy the MBR from sda to sdb: (DONT USE THIS COMMAND UNLESS YOU KNOW WHAT YOU ARE DOING)

dd if=/dev/sda of=/dev/sdb bs=512 count=1

it's important to get the /dev/sda and sdb right, because else, you will make your system unbootable (recoverable by a livecd though )
What you just did is copy 512 bytes from sda to sdb. these 512 bytes contain the partition table and your grub (bootloader)
(if=<input "file> of=<output file> bs=<block size> count=<how many blocks you want to copy>)

2. now you make the filesystems on sdb just like on sda (ext3, jfs whatever you have on sda)

3. now you can mount sda1, sda2 and so forth

4. rsync everything (everything) onto the correct partitions

<EDIT>
whoops, i totally forgot:

5. your fstab and grub configs need to be updated for the new UUIDs

</EDIT>

try it out by removing sda from your computer (temporarily) and trying to boot

TADAA

cheers Barde

Last edited by Heller_Barde (2009-05-20 06:44:50)

schuay · 2009-05-20 07:11:06

correct me if i'm wrong, but with cp/rsync you will be missing files like /dev/null, /dev/console, etc....

ataraxia · 2009-05-20 13:32:40

So, I feel I have to respond somehow to this, so as not to seem ungrateful for the nice responses, but none of these answer the question I actually have. I think the typical Arch-user's DIY attitude goes a bit too far, sometimes.

I know how to copy the files over, how to change the UUIDs, how to install GRUB, and how to create the 3 magic device nodes (null, console, and zero, for future reference). Just about any of these suggestions would be fine for a one-time or once-in-a-while process, but I want something that's painless enough to use as a daily backup routine. There's also the problem that they don't handle well the case where I don't want to take over the entire backup disk, and especially don't handle odd cases such as GPT disk format, where you shouldn't install boot code into the partition table area. They also aren't designed for the case where the backup device is an external disk which isn't intended to be remounted internally when disaster strikes, or the case of laptops where you don't want to have to remove /dev/sda from your machine to test.

Basically, it amazes me that apparently nobody has written software yet to automate what's probably a very common desire. Google didn't find anything of the sort, so I came here to ask actual human beings. (Yet another case where Google is not wrong )

Where I'm at now is to rsync everything and create the 3 device nodes, and then stop. This gets me to the 90% mark, is simple enough to be foolproof, and makes restoring the same as backing up. If there's no professional-feeling way to do the rest, I'll stop there.

None of this post is intended sarcastically, and please don't take offense at my laundry-list of complaints for the suggestions given. I think my posts could stand to be a little clearer if a paid sysadmin with 11 years of Linux background can gather responses that say things like "use cp" and "DONT USE THIS COMMAND UNLESS YOU KNOW WHAT YOU ARE DOING"

brisbin33 · 2009-05-20 14:44:06

ataraxia i feel your pain. there's nothing better for the job you describe than using SuperDuper! on my macbook. on linux, i've resorted to a two bash scripts: one to backup and one to restore. i've resigned myself to backing up /etc /usr /var /home individually via rsync along with a pacman -Qe listing. then if i have a failure i need to reinstall to a new drive, reinstall packages from the list, then restore those folders only via rsync again (i can then use the clean fstab / menu.lst to avoid UUID issues) this is mostly handled in the restore script.

i really think that's as good as it gets. i'm mainly posting here to [commiserate and] keep an eye on the thread incase a SuperDuper!-like solution emerges in linux.

best of luck.

karol · 2009-05-20 18:30:22

@brisbin33
Any particularly magical rsync settings?

ataraxia wrote:

Basically, it amazes me that apparently nobody has written software yet to automate what's probably a very common desire.

W/ my 11 years of using PCs I've never had much hw problems. It's not my job, so of course I have a different perspective, but here things tend to work and *I* am the weakest link (rm -r anyone?).

Last edited by karol (2009-05-20 18:30:42)

brisbin33 · 2009-05-20 18:38:10

karol, backup and restore; the restore one is a little less readable and definitely tailored for my needs. the backup is more general and could probably be easily tweaked.

Heller_Barde · 2009-05-20 19:19:56

ataraxia wrote:

and "DONT USE THIS COMMAND UNLESS YOU KNOW WHAT YOU ARE DOING"

actually, i was afraid that someone (whether that is you or someone else coming here) would not be careful

but even if i underestimated you, (shame on me) my suggestion still stands after your "complaints" (in lack of a better word)

after all the setting up is done, a regular rsync is all that is needed (with an appropriate file exclusion list of course) I think that is quite easy, no?

cheerio
Barde

Last edited by Heller_Barde (2009-05-20 19:21:45)

brisbin33 · 2009-05-20 19:30:10

Barde, i still think you're missing the point (if ataraxia doesn't mind me speaking on his behalf):

on a Mac
plug in usb hdd
click a button
wait

you now have a bootable backup that is an _exact_ copy of your running system

some time later...

zomg! complete hdd crash!
plug in usb hdd
reboot
click a button
wait
reboot

you're now up and running like it never happened

no amount of rsync scripting is going to accomplish _that_ in linux. period. i'm sorry.

Last edited by brisbin33 (2009-05-20 19:30:47)

ataraxia · 2009-05-20 19:46:43

brisbin33 wrote:

Barde, i still think you're missing the point (if ataraxia doesn't mind me speaking on his behalf):
on a Mac
plug in usb hdd
click a button
wait
you now have a bootable backup that is an _exact_ copy of your running system
some time later...
zomg! complete hdd crash!
plug in usb hdd
reboot
click a button
wait
reboot

you're now up and running like it never happened
no amount of rsync scripting is going to accomplish _that_ in linux. period. i'm sorry.

Complete agreement. You can speak for me anytime if you're going to do it this well.

karol · 2009-05-20 22:34:35

@brisbin33
Thanks for the scripts, I didnt't see them in your dotfiles / scripts ;-)

brisbin33 wrote:

no amount of rsync scripting is going to accomplish _that_ in linux. period. i'm sorry.

How much would it cost to hire sb to whip-up an app that makes that Mac-like backup at the swing of a magic wand^H^H^H^H wiimote? Easy enough? :-D

Last edited by karol (2009-05-20 22:35:09)

tkdfighter · 2009-05-20 23:01:48

You could just take the easy route and use the Gentoo Stage4, which is essentially just some instructions to make a tarball from which you can then recover according to the guidelines

brisbin33 · 2009-05-21 13:34:12

tkdfighter, i know you said that with a wink but here's the rub:

from their [very nice] installstage4.sh:

echo you need to check your fstab and install grub or lilo then
echo all should be well

tkdfighter · 2009-05-21 17:58:05

Well, look at the script. You aren't installing a master boot record, that's why it tells you to install grub or lilo. Alternatively, do this:

dd if=/dev/(your_disk) of=mbr.save count=1 bs=512 # backup
dd if=mbr.save of=/dev/(your_disk) # restore

Taken from http://en.gentoo-wiki.com/wiki/Custom_S … _table_tip .

Heller_Barde · 2009-05-21 18:51:38

brisbin33 wrote:

Barde, i still think you're missing the point (if ataraxia doesn't mind me speaking on his behalf):
<snip/>
no amount of rsync scripting is going to accomplish _that_ in linux. period. i'm sorry.

now, i don't try to be a smart-ass
but i still think that if you actually clone a harddisk (with dd or something similar) and afterwards keep the data up to date with rsync, it would actually really just work like you describe.

the preparation may not be as easy as one click like on mac, but it still should boot up just fiine

cheers
Barde

darthaxul · 2009-05-23 17:23:12

It really depends on how you want to replace the broke system, either with 1 full image or a series of images. You might just want to look into a raid-0, be much easier.

Arch Linux

#1 2009-05-20 00:47:49

how to create a bootable clone?

#2 2009-05-20 02:06:06

Re: how to create a bootable clone?

#3 2009-05-20 02:46:13

Re: how to create a bootable clone?

#4 2009-05-20 06:35:01

Re: how to create a bootable clone?

#5 2009-05-20 06:41:00

Re: how to create a bootable clone?

#6 2009-05-20 07:11:06

Re: how to create a bootable clone?

#7 2009-05-20 13:32:40

Re: how to create a bootable clone?

#8 2009-05-20 14:44:06

Re: how to create a bootable clone?

#9 2009-05-20 18:30:22

Re: how to create a bootable clone?

#10 2009-05-20 18:38:10

Re: how to create a bootable clone?

#11 2009-05-20 19:19:56

Re: how to create a bootable clone?

#12 2009-05-20 19:30:10

Re: how to create a bootable clone?

#13 2009-05-20 19:46:43

Re: how to create a bootable clone?

#14 2009-05-20 22:34:35

Re: how to create a bootable clone?

#15 2009-05-20 23:01:48

Re: how to create a bootable clone?

#16 2009-05-21 13:34:12

Re: how to create a bootable clone?

#17 2009-05-21 17:58:05

Re: how to create a bootable clone?

#18 2009-05-21 18:51:38

Re: how to create a bootable clone?

#19 2009-05-23 17:23:12

Re: how to create a bootable clone?

Board footer