How to securely "dd" four partitions onto a single USB Flash?

sharris · 2010-06-18 07:50:43

Hello again everybody,

I posted something about this in my previous thread because I didn't want to do back-to-back threads but I erased it because it was off track. I just need to do one more thing which should keep me busy for quite a while.

I have a 32GB Corsair USB Flash. I need to copy 4 partitions from my hard drive -- sda1, sda2 sda15 and sda16. Below are the exact size of each partition and the sizes will never change... After everything is in tack my goal is to learn how to build routers, firewall, etc for all four OS's back to back, excluding pcbsd, than optimize like crazy. I put a lot of thought into this and concluded, this kind of setup will make it painless and allow me to keep things in one place and well organize until I know what I'm doing and how I want to do it.

Primary 1: FreeBSD AMD-64 - 10GB
Primary 2: FreeBSD i386 - 10GB
Primary 3: pcBSD - 40GB
Logical sda15: Arch-Linux core-64 - 4GB
Logical sda16: Arch-Linux core-i386 - 4GB

Sda1  = 10GB - exactly 10,742,183,424 bytes
Sda2  = 10GB - exactly 10,742,215,680 bytes *32256-b larger.  fdisk bug i guest or the MBR thing for P1 ???  
Sda15 =  4GB - exactly  4,301,789,184 bytes
Sda16 =  4GB - exactly  4,301,789,184 bytes

My question is how would I use skip in dd to save all four partitions on a single Flash Drive with the possibility of never overwriting one another? Than, how would I restore them individually when needed?

I'm not sure if my calculations or setup below is 100% correct but this is all I have for now to give you an idea of what I'm trying to do. The only thing I know for sure is, if I am off by a hair I lose everything all at once.

................................... [b]1[/b]
Copy Partition-1 to Flash
dd if=/dev/sda1 of=/dev/sdb1 bs=1M
................................... [b]2[/b]
10742183424 P1 skip
10742183424 TOTAL

Copy Partition-2 to Flash (use skip)
dd if=/dev/sda2 of=/dev/sdb1 skip=10742183424 bs=1M
................................... [b]3[/b]
10742183424 skip P1
10742215680 skip P2
21484399104 TOTAL

Copy Partition-15 to Flash (double skip)
dd if=/dev/sda15 of=/dev/sdb1 skip=21484399104 bs=1M
................................... [b]4[/b]
10742183424 skip P1
10742215680 skip P2
 4301789184 skip P15
25786188288 TOTAL

Copy Partition-16 to Flash (triple skip)
dd if=/dev/sda16 of=/dev/sdb1 skip=25786188288 bs=1M

That's it. I'll be lucky if this is correct.

Thanks in advance

tomk · 2010-06-18 08:28:57

I would't use dd for this myself - instead, I'd suggest setting up the required partitions/filesystems on the usb device, and copying everything over. No worries about being off by a hair.

You could also look at partimage, might be useful.

sharris · 2010-06-18 10:39:14

Hi tomk,

That was only a joke, hehe. I would never worries about being off even by a micro-hair once I'm sure I got the correct numbers and command sequence. If someone will show me the way, I can begin testing. I like your idea but once you make up your mind to learn one way, it would be weak to replace it with something difference because somewhere down the line you'll be asking the same question over again for something else. Your suggestion would be a bonus if I knew how-to. So that would be another question before receiving an answer to the first. I am a noob right now.

"I wouldn't use dd for this myself - instead, I'd suggest"

Do this mean you're not sure how. Like the man lost in the desert, dying of thrust, he ran into an old prospector who offered him some ice cold water. The man turn to him and said "no thank you, I rather have a Budweiser" and continue his journey.

I forgot what Super Bowl commercial that was. Out of the late 80's I think. Could have been the Chicago-Bears. Whoever, it was but it was the best

Thanks tomk

Last edited by sharris (2010-06-18 10:47:47)

tomk · 2010-06-18 11:47:38

OK - hope you find that Budweiser.

/me keeps the Guinness for himself.

sharris · 2010-06-18 13:47:36

It read so good, I nearly dump my plan and was ready to install... than I got to this and it broke my heart.

The GPT layout .... Unfortunately, Windows for PC cannot be installed on a disk using the GPT layout.

It's good to know Partitions Commander will still works and I don't want to have problem living with DOS. I choose Arch-Linux so I can use its core functions to do everything possible with-out adding anything. I think GPT works like a dd script in the end. BTW, I don't drink but I'm glad you have a since of humor. Moderatorating don't seem easy.

Trent · 2010-06-18 15:25:25

sharris wrote:

I like your idea but once you make up your mind to learn one way, it would be weak to replace it with something difference because somewhere down the line you'll be asking the same question over again for something else.

You're asking how to drive in a screw with a hammer, and you're saying, "I don't want to learn how to use a screwdriver because I know one day I will need to use a hammer! Teach me to use a hammer!" dd is not the right tool for the job, just like a hammer is not the right tool to drive in a screw. Ask smart questions.

Apart from that -- sda15 and sda16? How many partitions do you have?

falconindy · 2010-06-18 15:46:47

Unless I'm mistaken, it doesn't seem like you need to access the data (except by partition at a time) from the flash drive. Why bother with all the calculations when you could just dd the partition to a file on the flash drive? Make a single partition on the flash drive and for each partition...

dd if=/dev/sdxy of=/path/to/flash/drive/sdxy.dev bs=1M

You can reverse the operation if/when you want to restore.

lilsirecho · 2010-06-18 17:25:32

A word to the wise...don't put all your eggs in one basket!!!!

sharris · 2010-06-18 21:35:20

Trent wrote:

You're asking how to drive in a screw with a hammer, and you're saying, "I don't want to learn how to use a screwdriver because I know one day I will need to use a hammer! Teach me to use a hammer!" dd is not the right tool for the job, just like a hammer is not the right tool to drive in a screw. Ask smart questions.

From 100's of web-sites around the world "There is no copy more intact than dd which use the most lowest level UNIX tool design specifically for cloning entire partitions and hard drives. The only difference is, I'm trying to add my own script and once I get an A-OK than I can will get some hands-on experience and see for myself if what you say is true.

Apart from that -- sda15 and sda16? How many partitions do you have?

I have 21 with SWAP, but I plan to cut it down to 15 since I recently wiped all other OS except BSD and Arch-Linux. It's a long story why... I'll just say I own my own copies of Windows Vista, Windows-7 32 and Windows-7 64 and after a year of trying to fix them, I don't like them anymore. Only Win95 - XP had true usable multi-windows so the older they got the worse they got. I also deleted Fedora and Ubuntu. I just completed my Linux+ and Network+ classes. My only interest now is configuration of low-level networking tools, so I have no use for other OS's robbing me of my concentration.

falconindy wrote:

Unless I'm mistaken, it doesn't seem like you need to access the data (except by partition at a time) from the flash drive. Why bother with all the calculations when you could just dd the partition to a file on the flash drive? Make a single partition on the flash drive and for each partition...

You hit the nail on the head. "No fun intended". This is an much better idea than my.

copy:
dd if=/dev/sda15 of=/mnt/sda15.img bs=1M

falconindy wrote:

You can reverse the operation if/when you want to restore.

 Restore:
dd if=/mnt/sda15.img of=/dev/sda15 bs=1M

Yes ... Yes ... Yes ...
One question... PLEASE say _ _ _ !!!

1) Can I point to the device file and than the name of the img file on that device with-out mounting the device itself... like this:

copy:
dd if=/dev/sda15 of=/dev/sdb1/sda15.img bs=1M

restore:
dd if=/dev/sdb1/sda15.img of=/dev/sda15 bs=1M

The problem is I can't test like I want to because it might take like 2-3 days to wipe my 32GB Flash on my 2009 Quad-Core machine and that don't make no since. At this very minute since learning how-to I'm doing it under Arch right now and 11 hours has pass and it's still wiping. So I'm not happy with everything about dd. It could be faulty hardware far as I know. But I always wipe my disk so I'm ensured a clean start. I should have notrunc to wipe any possible Window virus but that takes a week than I still pull the plug. And here i am defending dd. Anyway, I think your way will be final but I still like to know if my numbers were on track or not. Skip is interesting. It's more about knowing how-to do stuff on your own than anything else.

dd if=/dev/zero of=/dev/sdb1 bs=1M

lilsirecho wrote:

A word to the wise...don't put all your eggs in one basket!!!!

I guarantee you, I don't plan to. That's why I do FreeBSD and Arch-Linux and planning for a decent way to back it all up. I been tricked by my own bad main-system backup before. Lucky for me my life work was on every spare CD in the house and basement. But I was still stuck re-installing everything, including much custom stuff that some I forgot how to do. That hurts

So after today, my many Flash back-ups will be safe. That's what this whole thread is all about ... The only problem was that I choose an more exotic way of backing up that seem to have cause a spark or two from coast to coast and now, around the world.

fsckd · 2010-06-18 21:51:45

sharris wrote:

Trent wrote:
You're asking how to drive in a screw with a hammer, and you're saying, "I don't want to learn how to use a screwdriver because I know one day I will need to use a hammer! Teach me to use a hammer!" dd is not the right tool for the job, just like a hammer is not the right tool to drive in a screw. Ask smart questions.
From 100's of web-sites around the world "There is no copy more intact than dd which use the most lowest level UNIX tool design specifically for cloning entire partitions and hard drives. The only difference is, I'm trying to add my own script and once I get an A-OK than I can will get some hands-on experience and see for myself if what you say is true.

Its very good at making byte by byte copies of contents but sucks if you care about meta info, or in your case, the context of the stuff being copied.

1) Can I point to the device file and than the name of the img file on that device with-out mounting the device itself... like this:
copy:
dd if=/dev/sda15 of=/dev/sdb1/sda15.img bs=1M
restore:
dd if=/dev/sdb1/sda15.img of=/dev/sda15 bs=1M

No, you need to mount first. Care to guess why?

sharris · 2010-06-19 00:18:07

No, you need to mount first. Care to guess why?

Well for one thing I made an mistake editing falconindy code above. Also, I forgot some of what I learn in Linux+. It takes more than 16-weeks of cramming to be an expert. Now I remember why I needed to do this with SKIP and yes I care to guest why, thank you. Back in class when I first tried to dd a file with-out mounting it, it gave an error something like "This is not a directory". But if I dd the device only it works with-out mounting. From that day I never looked back at dd'ng files, so I forgot.

It was nice to think falconindy had my solution. So it back to SKIP and bad treatment. What get me is, I read the documents and something just don't click. So I go to many samples sites and most would say skip=25786188288 represent BLOCK size, while others make it seems like it BYTE size. I don't know how to count blocks so I posted thinking BYTE and hope for a reply to give a solution, a clue or a link.

fsckd

Its very good at making byte by byte copies of contents but sucks if you care about meta info, or in your case, the context of the stuff being copied.

Are you telling me when we use dd to clone a partition it don't copy ALL data. If so would you mind pointing me the source that carry these type statements. If not, it's not good to post mis-leading statements.

falconindy · 2010-06-19 00:34:14

sharris wrote:

1) Can I point to the device file and than the name of the img file on that device with-out mounting the device itself... like this:
copy:
dd if=/dev/sda15 of=/dev/sdb1/sda15.img bs=1M
restore:
dd if=/dev/sdb1/sda15.img of=/dev/sda15 bs=1M

Of course you can't. These commands make no sense. I think your fundamental understanding of the relationship between block devices and partitions is flawed. /dev/sdb1 is a block device, not a directory. You can't treat it like a directory hoping that you can write directly to it. There needs to exist an abstraction layer (or multiple layers) between the block device and the user in order for you to accomplish this. What's the issue with mounting the partition containing the dd'd .img files first?

fsckd · 2010-06-19 01:19:11

sharris wrote:

Its very good at making byte by byte copies of contents but sucks if you care about meta info, or in your case, the context of the stuff being copied.
Are you telling me when we use dd to clone a partition it don't copy ALL data. If so would you mind pointing me the source that carry these type statements. If not, it's not good to post mis-leading statements.

I believe you have misunderstood me. What I made was a rather off-hand remark for completion. It should be obvious from a quick perusal of dd's man page that it will not preserve meta info such as name, timestamps, access control, and so on. That is the meta info to which I refer. In regards to your issue, copying the partition data will not copy any info about it in the partition table. You will need a separate dd command to backup the MBR, which is what I assume your machine is using. The tool dd is for copying the contents of files. It works for disk drives and partitions because the OS presents them as files. Very convenient, no?

Please be aware that we are volunteers and, while we have our fun, we try to maintain some professionalism. We will not mislead you or try to trick you into doing something harmful. We can make mistakes and it is good to call us out on that. But please do not assume our intent is nefarious. k?

sharris · 2010-06-19 01:23:52

Of course you can't. These commands make no sense. I think your fundamental understanding of the relationship between block devices and partitions is flawed. /dev/sdb1 is a block device, not a directory. You can't treat it like a directory hoping that you can write directly to it. There needs to exist an abstraction layer (or multiple layers) between the block device and the user in order for you to accomplish this.

I understand all of that but cannot say it as good as you just did. But something is missing .... hmmmmmm

My question is how would I use skip in dd to save all four partitions on a single Flash Drive with the possibility of never overwriting one another? Than, how would I restore them individually when needed?
I'm not sure if my calculations or setup below is 100% correct but this is all I have for now to give you an idea of what I'm trying to do. The only thing I know for sure is, if I am off by a hair I lose everything all at once.

Now I remember, you also are one who never said anyhing completely related to the original question?

"What's the issue with mounting the partition containing the dd'd .img files first"

I forgot what the issue. I just know it's in the pass and I know what I want to do today. If something is possible in a command-line system, I want to try it. SKIP is possible. How about an example based on what I presented. Arch and FreeBSD is for want-to-be low lever coder. Arch Desktop is for others and we don't question them of why they made that choice or try to discourages them just because they did not follow our suggestion and im not going to gang up on them in an effort to help evade their original questions.

sharris · 2010-06-19 01:47:09

Please be aware that we are volunteers and, while we have our fun, we try to maintain some professionalism. We will not mislead you or try to trick you into doing something harmful. We can make mistakes and it is good to call us out on that. But please do not assume our intent is nefarious. k?

I don't mind you guys tripping and having some fun but I did not question about what dd can't do, or dd and files, or how to ask smarter questions. This is all I asked ....

My question is how would I use skip in dd to save all four partitions on a single Flash Drive with the possibility of never overwriting one another? Than, how would I restore them individually when needed?
I'm not sure if my calculations or setup below is 100% correct but this is all I have for now to give you an idea of what I'm trying to do. The only thing I know for sure is, if I am off by a hair I lose everything all at once.

Now I'm only having fun chit-chatting with a dash of fuss with you guys. I not upset. I'm Just wore out from answering all the questions and explaining why I ask such a highly technical question for a noob, and ever one of you guys know I posted ONE MEAN block of code. But maybe none of us can read by nature and can tell if it works or not by glance.

sharris · 2010-06-19 02:20:17

Now I see what's going on. Tomk provided the next best thing posibble to my question because it take some serious coding to do what I'm asking or you must simply use a program off the self. You guy thought I was picking on tomk and ran to his rescue. We were, or at lease I was just having fun and showing some personality and my joke had good timing.

Anyway, don't worry about it, I'll write it in Ark, C or Perl one day soon. Under thirty lines should do. I forgot that is one of the things I must set shop for. I use to love coding for Windows but now it's a new ball-game. I got to learn UNIX. Sorry for torturing you guys. That stuff is heavy... When I write it I'll post it here

Have a Great Day

falconindy · 2010-06-19 02:24:36

sharris wrote:

My question is how would I use skip in dd to save all four partitions on a single Flash Drive with the possibility of never overwriting one another? Than, how would I restore them individually when needed?
I'm not sure if my calculations or setup below is 100% correct but this is all I have for now to give you an idea of what I'm trying to do. The only thing I know for sure is, if I am off by a hair I lose everything all at once.
Now I remember, you also are one who never said anyhing completely related to the original question?

Perhaps you fail to understand my proposed solution and actually trying it on a smaller scale would be an effective experiment. dd'ing a partition to a file means that you do not need to know the start and end of the partition. You don't need to use the 'skip' argument of dd. You simply image the block device and the result is a portable file representation.

But carry on. Go and write your thirty lines when 1 line would suffice.

sharris · 2010-06-19 02:42:32

But carry on. Go and write your thirty lines when 1 line would suffice.

This is getting ridiculous. Prove it. If it's so easy and you been holding back all along, post your one line example today. I'm not going to cry if it takes you more... Just do it.

fsckd · 2010-06-19 02:58:54

use your imagination: pmount /dev/sdb1 && for i in 1 2 15 16 ; do dd if=/dev/sda$1 of=/media/sdb1/sda$1.img bs=1M ; done

warning: use the above command at your own risk

Edit: you are demanding stuff from us, please stop demanding, asking is fine, demanding is not

Last edited by fsckd (2010-06-19 03:00:29)

falconindy · 2010-06-19 03:24:25

sharris wrote:

But carry on. Go and write your thirty lines when 1 line would suffice.
This is getting ridiculous. Prove it. If it's so easy and you been holding back all along, post your one line example today. I'm not going to cry if it takes you more... Just do it.

Sure. Scroll up the page to post #7 and look at what I already gave you. Ridiculous is how much you're overthinking this simple task.

sharris · 2010-06-19 04:21:10

Stop griping and post some real working examples based on the original question with restore code. I will run the risk of testing and correcting if needed. So please add comments so I can figure on adjustments. It's time to sleep now. Good night

fsckd · 2010-06-19 04:59:32

sharris wrote:

Stop griping and post some real working examples based on the original question with restore code. I will run the risk of testing and correcting if needed. So please add comments so I can figure on adjustments. It's time to sleep now. Good night

Fine. This seems to be going completely over your head. I'll try to explain it as simply as I can. You do not use skip to copy a partition with dd. You do not use skip because there is nothing to skip. The partitions are already separated for you by the OS. When you copy, for example /dev/sda15, you start from the first byte. Why? Because the OS has kindly started partition 15 from the beginning of /dev/sda15. Partition 16 starts from the beginning of /dev/sda16. Look at what you wrote: dd if=/dev/sda15 of=/dev/sdb1 skip=21484399104 bs=1M . It will skip the first 21 GB of partition 15 and error out because partition 15 is only 4 GB. (It may not error out but start copying garbage from elsewhere. Also, it's actually TB, bs sets the block size to 1 MiB and skip jumps over 21484399104 blocks or 21484399104 MB which is over 21 TB.) The correct line would be dd if=/dev/sda15 of=/dev/sdb1 seek=21484399104 . When you restore, you would use skip in place of seek and change if and of accordingly. Happy? This would not be a robust setup. If your partition layout changes in any way, your restore process would not work. Hence Tomk's advice. You will also restrict your "backup" to a single device. Hence falconindy's advice.

Last edited by fsckd (2010-06-19 05:01:57)

tomk · 2010-06-19 07:06:49

Closing this thread.

sharris - feel free to start another thread if you wish, but be aware that your attitude in this thread leaves a lot to be desired. There are guidelines concerning the behaviour expected in this forum - I suggest you read them.

Arch Linux

#1 2010-06-18 07:50:43

How to securely "dd" four partitions onto a single USB Flash?

#2 2010-06-18 08:28:57

Re: How to securely "dd" four partitions onto a single USB Flash?

#3 2010-06-18 10:39:14

Re: How to securely "dd" four partitions onto a single USB Flash?

#4 2010-06-18 11:47:38

Re: How to securely "dd" four partitions onto a single USB Flash?

#5 2010-06-18 13:47:36

Re: How to securely "dd" four partitions onto a single USB Flash?

#6 2010-06-18 15:25:25

Re: How to securely "dd" four partitions onto a single USB Flash?

#7 2010-06-18 15:46:47

Re: How to securely "dd" four partitions onto a single USB Flash?

#8 2010-06-18 17:25:32

Re: How to securely "dd" four partitions onto a single USB Flash?

#9 2010-06-18 21:35:20

Re: How to securely "dd" four partitions onto a single USB Flash?

#10 2010-06-18 21:51:45

Re: How to securely "dd" four partitions onto a single USB Flash?

#11 2010-06-19 00:18:07

Re: How to securely "dd" four partitions onto a single USB Flash?

#12 2010-06-19 00:34:14

Re: How to securely "dd" four partitions onto a single USB Flash?

#13 2010-06-19 01:19:11

Re: How to securely "dd" four partitions onto a single USB Flash?

#14 2010-06-19 01:23:52

Re: How to securely "dd" four partitions onto a single USB Flash?

#15 2010-06-19 01:47:09

Re: How to securely "dd" four partitions onto a single USB Flash?

#16 2010-06-19 02:20:17

Re: How to securely "dd" four partitions onto a single USB Flash?

#17 2010-06-19 02:24:36

Re: How to securely "dd" four partitions onto a single USB Flash?

#18 2010-06-19 02:42:32

Re: How to securely "dd" four partitions onto a single USB Flash?

#19 2010-06-19 02:58:54

Re: How to securely "dd" four partitions onto a single USB Flash?

#20 2010-06-19 03:24:25

Re: How to securely "dd" four partitions onto a single USB Flash?

#21 2010-06-19 04:21:10

Re: How to securely "dd" four partitions onto a single USB Flash?

#22 2010-06-19 04:59:32

Re: How to securely "dd" four partitions onto a single USB Flash?

#23 2010-06-19 07:06:49

Re: How to securely "dd" four partitions onto a single USB Flash?

Board footer