Help with rescuing data from faulty disk

pablox · 2014-07-29 06:03:20

Sorry the long wall of text, I wanted to explained everything as detailed as possible to not mess with the chronological order (TL; DR down below!)

Ok, here's the thing. I have two hard drives, a ssd [sda] (for the OS [arch linux, ubuntu and win7], munchkin 240gb) and a sata 3 one ([sdc] wd, 1 tb) that I use for everything else. A couple of weeks ago I found there were progressive, unexpected and random behaviour particularly with windows, random freezes, no bsod, ecc... I finally found that the second hard drive was the culprit so I decided stop using a buy a new disk (while sending the faulty to RMA, since it's barely one year old, and I don't do nothing out of ordinary), I got a seagate with 1 tb.

I honestly didn't want to mess too much about the data and wanted to save the most without wasting time, so I decided to try to clone the whole disk with dd according to the wiki (Disk Cloning). I imagined that the "bad sectors" were from the ntfs partition, so I went with that. Clearly I could get corrupt files in linux too, but my guess was that the most important data was safe, because I had been using it for the last couple of days without problems. The process ended with a lot of I/O errors, but in the end it showed the same number of `in` and `out`. I regenerated the UUID for the new cloned disk and each partition, modified the fstab and restarted (fingers crossed).

I could log into arch linux without problems, used it for a while, restarted a second time (that windows habit that I still found hard to overcome) and everything was okay (yes I made sure I was actually on the new disk). Next was windows time, which to my surprise worked fine too, I even played a while. After patting myself in the back, I rebooted the machine thinking in how much work I would like to have done today, and there were an error that apparead on the login screen, just a ms before I pressed enter, I think I read it was referring to another partition of sda. The system got stuck in the middle of the "log in screen" and loading x with a black screen, nothing else. Waited a while, losing hope I pushed the RESET button.

Now, it was stuck during the boot up process, after "killing rf radio" or something. It was obviously stuck somewhere else... (that message had appeared there since I built the pc) it read though that the /var partition was clean, same with the root partition, next I tried with "recovery" kernels, and each other kernel I had still in grub; but the problem persisted. Tried with ubuntu, had to change fstab again, rebooted, tried to enter, stuck on the loading screen with the big purple background. I couldn't get anything else past that, neither to get a decent log.

I don't know why (or maybe because I was desperate) I entered the uefi/bios settings and found that the time was localtime instead utc.... weird I thought and I blamed windows (though, I had the registry hack working for a while) and imagined that I updated something that removed that *fixed* the blessed fix. Since I remember having issues with partitions not being correctly mounted once because of time issues, I thought it was a problem with the time... nope, nothing changed.

I tried again with the recovery kernel of the ubuntu install, I could drop on the root shell, and tried to fix the clock time after remounting, same with mounting the /var partition (ubuntu doesn't use that one), it threw that "bad superblock error", but fsck ran saying the filesystem was clean and it was mount normally later. Somewhere between, I mounted the home partition, no errors, gave a quick navigation. I tried to install ntpd to follow the "troubleshoot" from the Time arch wiki page, but HA, no internet.

Rebooted again, and from the menu from "ubuntu recovery", allowed me to enable networking. Quite nice I thought, and pressed enter. It politely informed that I needed to mount the system (root partition) in read/write instead read-only. I happily said yes (I already had remounted the ubuntu root as rw, and worked ok), but stopped at the home partition taking about 10 minutes in "phase 2", suddenly started throwing a lot of errors, about missing inodes, files pointing nowhere, etc. It put a lot of files into lost+found directory. Oh, there it was... I thought, let's go into Arch Linux then...

Now the system booted up quite fast, and I was welcomed by the black screen asking coldly for my login information, I wrote the credentials, pressed enter, aaaaand PAH, "/home/pablo" change directory failed: No such file or directory. HA, funny I thought, lsblk, the partition was mounted, ls -l /home/... WHAT? the folder wasn't there.

I looked for the lost+found folder inside the home directory, and it had a lot of directories and files called DXXXXXX.RCN, and IXXXXXX.RCN and some symbolic links. The directories had files inside, the IXXXXX are actually files, ecc.

Right now, I'm lost... I'm not scared about the data, because it's still living happily (?) on the bad disk, but I still need to recover it, so I can filter it later allowing to write zeros to the fault disk and send it for a replacement. Is there any chance to recover the data? (I rather not going into around 100 files/directories with cryptics names), or maybe use another method to "recover" the data from the previous disk? using cp or rsync? Any good experience or advice on this?

PS.- Sorry for the epic story, I hope I don't scare readers or other fellow members.

TL; DR

+ I have 2 disks (ssd [OS {win7, arch and ubuntu}] and sata3 [data])
+ Sata3 disk started to fail on windows (users folder it's on the sata3), arch/ubuntu never had a problem
+ I bought a new one, and I'm trying to send the old one for a RMA
+ Clone the disk according to the wiki and rebuilt the UUID
+ Change fstab to use the new cloned disk
+ Worked fine for a couple of times
+ Used windows, played, everything fine
+ Entered arch, now I couldn't it got stuck without showing any errors
+ Tried every kernel and recovery image without success
+ With Ubuntu I could drop into a root shell
+ Mounted home partition, no problems
+ Tried to activate networking, started to run fsck and threw a lot of errors (I remember I read something about "offset" not correcting) and moved files to lost+found directory
+ Arch Linux started, but I couldn't log because /home/pablo doesn't exist anymore and it seems everything lives now in the lost+found directory with weird names like DXXXXXX and IXXXXXXX

Aftermath: I have the original data, safe (?) on the bad disk (like I said, I never had any problem in my home partition). I really don't want to be nit picky about what I can save, I'll be happy to save as much as I can, but I don't mind losing corrupt files (that's why I thought cloning with filling with zeros would be enough). I just want to ditch the bad disk and send it to wd for a replacement.

End of TL; DR

2014/07/29: Added a TL; DR section

Last edited by pablox (2014-07-29 13:55:46)

R00KIE · 2014-07-29 07:35:05

The first rule of data recovery is to stop using the affected disk(s), if you don't your chances of successful data recovery can go down. The second rule is to not do any writes to the affected disk(s), such as fsck, when trying to recover your data. If the data is really important and can't be recreated you probably should look into professional data recovery services.

If trying to recover the data yourself get a big disk, at least as big as the affected disk(s), and do a full image with ddrescure. Then work from the image while keeping the image read only. If you want to do tests on the image or try any writes do a copy of the image and work on that.

stqn · 2014-07-29 10:14:40

That’s *really* a big wall of text and I can’t quickly see what your problem is.

As R00KIE wrote, use ddrescue to recover your data. Then my advice would be to just install a clean OS on your new drive (after testing the new drive with badblocks), then copy your salvaged personal data on it.

(Note that there are two different packages: extra/ddrescue and community/dd_rescue… And I don’t remember which one I used.)

Edit: I used dd_rescue to recover each file one by one with the following script, but it might be better to recover the whole disk or partition with ddrescue instead.

#!/bin/bash
set -e

target="/media/newdisk/home/user"
src="/media/brokendisk/home/user"

cd "$src"
find . -type d -exec mkdir "$target/{}" \;
find . -type f -exec dd_rescue -v -l "$target/{}.ddrescue.log" "{}" "$target/{}" \;

Last edited by stqn (2014-07-29 10:32:15)

R00KIE · 2014-07-29 13:29:44

I have always used ddrescue and so far I have no complaints. I can't comment about dd_rescue as I have never used it. What I can add is that I prefer full disk images because those are more fool proof (read: less chance to forget to copy something). I'd say a full disk copy is faster and easier on the bad disk than copying file by file (no random seeks).

What stqn says about using badblocks to test the new disk is good advice, I do use it to do the full write+read test on new disks before I put them to use, just to make sure the new disk doesn't have manufacturing defects.

pablox · 2014-07-29 13:41:14

Thanks both for your answers . Yeah, I know it's a big wall of text , I though have fun writing though (maybe because it was the only way to express my frustration).

The thing is that I'm not using the bad disk, last time I used it was two days ago when I tried to clone it, with lot of I/O errors. Since the ntfs partition is working fine, I will use ddrescue to recover my old home partition tomorrow and I will come back to post the results.

(I'll add a TL ;DR to the main post btw) Done.

Last edited by pablox (2014-07-29 13:56:19)

pablox · 2014-07-29 14:00:10

R00KIE wrote:

I have always used ddrescue and so far I have no complaints. I can't comment about dd_rescue as I have never used it. What I can add is that I prefer full disk images because those are more fool proof (read: less chance to forget to copy something). I'd say a full disk copy is faster and easier on the bad disk than copying file by file (no random seeks).

Agree, that's why I thought cloning was enough, it seems I was wrong though, I should have used ddrescue.

R00KIE wrote:

What stqn says about using badblocks to test the new disk is good advice, I do use it to do the full write+read test on new disks before I put them to use, just to make sure the new disk doesn't have manufacturing defects.

That's a good idea.... I just have the disk, so I think I'm starting from zero again. How is that I could that? Should I write the disk with zeros? a lá

dd if=/dev/zero of=/dev/sdb # ?

R00KIE · 2014-07-29 16:02:53

pablox wrote:

That's a good idea.... I just have the disk, so I think I'm starting from zero again. How is that I could that? Should I write the disk with zeros? a lá
dd if=/dev/zero of=/dev/sdb # ? 

No need for that, I usually use 'badblocks -sw /dev/sdX' as root and let it run until it finishes. If it doesn't burp any errors you should be good to go. You do this on your _new_ disk, not the one which has your data.

Edit:
Clone the whole disk, just because now you can access a partition and retrieve your data it doesn't mean you can do it next time you try.

Last edited by R00KIE (2014-07-29 16:04:42)

stqn · 2014-07-29 16:04:17

pablox wrote:

Should I write the disk with zeros?

No need, just use badblocks in read/write mode…

pablox · 2014-07-30 03:53:25

Uhm, sorry... so I should go with:

1. Ran badblocks -sw /dev/sdb (new disk) <- afaik it will destroy the data I already cloned (right now it's not much use in those weird lost+found directory)
2. Instead cloning it again, clone it using ddrescue (or even better, the home partition that's the one that's giving problems; ntfs partition is running fine).

Like I told, I already cloned the disc with `dd if=/dev/sdc of=/dev/sdb bs=4096 conv=notrunc,noerror,sync` (it's not a typo, the "new disk", appeared on sdb), and it led me to the situation I'm standing right now. What good would do trying to clone again? (or maybe I misunderstood something).

stqn · 2014-07-30 08:00:17

Yes, badblocks -w will destroy the data. I’m sure that’s what the man page says anyway. You can use -n to do a non-destructive read-write test instead.

ddrescue was made to recover data, dd wasn’t. I you think what you’ve already got with dd is good enough, then you can keep it I suppose…

mich41 · 2014-07-30 22:17:54

I can see three possibilities:

1. The new HDD is broken as well.
2. The Linux partition contained a little garbage from old HDD's errors and that confused Linux and caused it to corrupt the partition further by writing more garbage to it.
3. The garbage from old HDD's errors contained on the Windows partition confused Windows and caused it to corrupt the Linux partition by overwriting it with garbage.

1. Rather unlikely if the HDD is new, but still worth checking. Install smartmontools and run smartctl -a /dev/new-hdd. On the "SMART attributes" list, check if any of "Reallocated_Sector_Ct", "Reallocated_Event_Count", "Current_Pending_Sector", "Offline_Uncorrectable" have RAW_VALUE other than zero or if the "SMART Error Log" contains any entries. If the attributes are all zero and the error log is empty, everything is fine.

2. Make new image of the old HDD. Make sure that Windows and Linux partitions don't overlap each other due to HDD read error. Run fsck on the Linux partition before mounting it to ensure that it doesn't contain errors which could lead to further errors. If it doesn't, it should be safe to mount and use it.

3. You said that Windows partition certainly contains error. Don't run Windows and don't mount this partition in Linux without running some NTFS checker (Windows tools on some other Windows system or ntfsck on Linux) and fixing all errors contained there. Alternatively, you can reinstall Windows on the new HDD and manually copy all the files you care about that are still readable from the old HDD. BTW, don't modify anything on the old HDD, as this would likely corrupt its other contents in the same way that modifying its copy corrupted other parts of its copy.

Last edited by mich41 (2014-07-30 22:30:11)

pablox · 2014-08-01 01:47:22

Ok... so I tested the new hdd with badblocks -sw, which didn't throw any errors. I made the "cloning" again, but with ddrescue, I ran it two times as is suggested in the info manual, even with retrying and all the magic it supposed it does.

I ran fsck.jfs -fv /dev/sdb4 (which is the device where my own partition was). and PAH. It started saying that there were corrupt data, and threw the a lot of errors:

File inode 4524 has been reconnected to lost+found
Directory inode 4527 has been reconnected to lost+found

148 directories reconnected to /lost+found
95 files reconnected to /lost+found

Then everything in my home directory was threw spitted into a lot of CHN files, which have the data, but honestly I don't feel like checking every file there >.<

I can came up, with two alternatives...

copy the files trough rsync, and just skipping the files with errors (I don't mind losing them), I just want to save everything as possible. Just check and copy the directories over the home directory D:

R00KIE · 2014-08-01 06:28:41

R00KIE wrote:

If trying to recover the data yourself get a big disk, at least as big as the affected disk(s), and do a full image with ddrescure. Then work from the image while keeping the image read only. If you want to do tests on the image or try any writes do a copy of the image and work on that.

You missed this part right? You never write to the backup image, fsck counts as writing to the image.

pablox · 2014-08-01 16:04:52

R00KIE wrote:

R00KIE wrote:
If trying to recover the data yourself get a big disk, at least as big as the affected disk(s), and do a full image with ddrescure. Then work from the image while keeping the image read only. If you want to do tests on the image or try any writes do a copy of the image and work on that.
You missed this part right? You never write to the backup image, fsck counts as writing to the image.

Hmm, no I read it... but I don't understand. I cloned the disk with ddrescue. I haven't touched or anything the faulty disk. I ran fsck in the copy before mounting, because I can't ran fsck if the disk is already mounted. The info manual on ddrescue states clearly that you must run fsck before trying to mount the rescued partition. But I understand that this fsck is giving more problems, because I actually used the rescued partition a couple of times before the system started to fail strangely.

I already rescued the partition again (I saved already everything for the ntfs partition), and I'm going to dd it, so I can have the original /home partition, a copy and another copy, and in the last one I'm going to start copying the files through rsync to an external disk (I have to study which are the best way to do it, I guess I'll do it like its a backup), while mounting it read-only. If that completes ok, I will repartition the new hd again, and starts from "fresh".

R00KIE · 2014-08-01 20:46:04

The thing is, after you do the backup you don't touch either the bad disk or the copy you just made of the bad disk, if you want to test things you do it in a copy of the copy, you always keep a copy that you don't touch (read only is ok). I can't make it easier to understand than this:

Wrong - what you are doing
1 - Make copy of bad disk.
2 - Make changes on the copy, things go wrong, new copy needed.
3 - Disk fails, copy not possible.
4 - Data loss

Right - how you should do it
1 - Make copy A of bad disk.
2 - Make copy B of copy A.
3 - Make changes on copy B , things go wrong, new copy needed.
4 - Make new copy B from copy A.
5 - Try different changes on copy B, ....
6 - Rinse and repeat until data can be recovered.

Arch Linux

#1 2014-07-29 06:03:20

Help with rescuing data from faulty disk

#2 2014-07-29 07:35:05

Re: Help with rescuing data from faulty disk

#3 2014-07-29 10:14:40

Re: Help with rescuing data from faulty disk

#4 2014-07-29 13:29:44

Re: Help with rescuing data from faulty disk

#5 2014-07-29 13:41:14

Re: Help with rescuing data from faulty disk

#6 2014-07-29 14:00:10

Re: Help with rescuing data from faulty disk

#7 2014-07-29 16:02:53

Re: Help with rescuing data from faulty disk

#8 2014-07-29 16:04:17

Re: Help with rescuing data from faulty disk

#9 2014-07-30 03:53:25

Re: Help with rescuing data from faulty disk

#10 2014-07-30 08:00:17

Re: Help with rescuing data from faulty disk

#11 2014-07-30 22:17:54

Re: Help with rescuing data from faulty disk

#12 2014-08-01 01:47:22

Re: Help with rescuing data from faulty disk

#13 2014-08-01 06:28:41

Re: Help with rescuing data from faulty disk

#14 2014-08-01 16:04:52

Re: Help with rescuing data from faulty disk

#15 2014-08-01 20:46:04

Re: Help with rescuing data from faulty disk

Board footer