[SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

archfluke · 2017-12-16 03:25:46

after using testdisk to recover a deleted Windows partition on my dual boot system, it for some reason, by default deletes all partitions unless marked otherwise. That said, I was wondering how i can recover from this fiasco. There was a thread here https://bbs.archlinux.org/viewtopic.php?id=151089 that unfortunately didn't get resolved. However, I did manage to get further than the poster of that thread by finding the LUKS headers using hexdump -C /dev/sdb4 | grep LUKS:

7d000200  4c 55 4b 53 ba be 00 01  61 65 73 00 00 00 00 00  |LUKS....aes.....|

So, thankfully (or at least, hopefully), testdisk didn't overwrite my LUKS header and simply did some funky partition shifting. Unfortunately, I don't know where to go from here and would appreciate any help in recovering this data.

I've been referencing this thread as well: https://ubuntuforums.org/showthread.php?t=1643334 and it mentions using a loop device but I wasn't able to find documentation that takes me step by step through the process, but then now I'm finding that one of the steps it advises is to use losetup, which I can't use since I get losetup: cannot find an unused loop device. I haven't been able to find documentation on loop devices the way I found documentation on installing archlinux on the wiki.

EDIT: I found that the reason why losetup wasn't working was because on my working install of archlinux, I had updated the kernel and hadn't rebooted. Rebooting allowed me to run losetup with no issues.

Thank you.

Last edited by archfluke (2017-12-17 19:55:13)

frostschutz · 2017-12-16 03:35:09

You can try to make a readonly loop device with data offset

losetup --find --show --read-only --offset $((0x7d000200)) /dev/sdb4

then see if you can luksopen it.

It's a very odd data offset, though.

archfluke · 2017-12-16 03:54:42

frostschutz: Thanks for the response! By the way, it was awesome seeing you systematically solve the issue in the other thread on ubuntuforums in a way that a n00b like myself can follow.

On the current issue, I was able to enter a passphrase. But when I run sudo mount /dev/mapper/luksrecover mnt, I get

mount: /home/qkay/mnt: unknown filesystem type 'LVM2_member'

Last edited by archfluke (2017-12-16 04:04:07)

R00KIE · 2017-12-16 13:51:00

You probably want to give us a better picture of how your partition layout looked before it got deleted/overwritten.

That said and from the error you got, try 'lvchange -ay' and then check if any volume groups got activated and check if you see your logical volumes.

archfluke · 2017-12-16 15:41:09

R00KIE: hello. Unfortunately, not knowing of testdisk's potential to be destructive and not knowing that it defaults to deleting all partitions, I did not have the foresight to backup the partition table's state prior to using testdisk. The output from running

fdisk -l /dev/sdb

outputs this:

Disk /dev/sdb: 3.7 TiB, 4000752599040 bytes, 7813969920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 80AA99CC-01CE-E346-BC42-1CD9AA654112

Device         Start       End   Sectors   Size Type
/dev/sdb1       2048    804863    802816   392M Microsoft basic data
/dev/sdb2     804864   1083391    278528   136M Microsoft basic data
/dev/sdb3    1083392 245972990 244889599 116.8G Microsoft basic data
/dev/sdb4  245972991 490862589 244889599 116.8G Microsoft basic data

Partition 4 does not start on physical sector boundary.

Partition 4 is the luks container and there was a bit of unallocated space on the disk prior to the incident. But I read elsewhere that I only need to know where the partition starts.

Running

lvchange -ay

outputs this:

  No command with matching syntax recognised.  Run 'lvchange --help' for more information.
  Nearest similar command has syntax:
  lvchange -a|--activate y|n|ay VG|LV|Tag|Select ...
  Activate or deactivate an LV.

One thing that I also noticed from the other thread I linked to in the OP was that the person who solved it says this:

After fixing the cylinder boundary issue, I read a tutorial on sfdisk, sent the output of sfdisk -d to a text file, and started manually editing the sector start and size for /dev/sda7, writing the changes, then using hexdump to see where the LUKS header was until it lined up with 0x00000000.

Three questions come to mind:
1) how did s/he fix the cylinder boundary issue?
2) what were the steps involved in "manually editing the sector start and size" for the partition to be rescued?
3) if I ran sfdisk -d for a partition that's 128GB, does that mean the output text file will at least be that large?

Side note: the person who had the issue on that other thread reports this:

root@ubuntu:/home/ubuntu# mount -o loop,offset=0x000f8200 /dev/sda7 /media/test
mount: unknown filesystem type 'crypto_LUKS'

I get the same error message when trying to mount the luks container similarly, which bears resemblance to the LVM2_member message I got.

frostschutz · 2017-12-16 15:46:59

that'd be vgchange -a y not lvchange ...

R00KIE · 2017-12-16 15:54:02

frostschutz wrote:

that'd be vgchange -a y not lvchange ...

Ah yes, that, I haven't used lvm in a bit of time and memory is a bit rusty.

archfluke · 2017-12-16 15:59:42

thanks. vgchange -ay outputs this:

  2 logical volume(s) in volume group "MyStorage" now active

which I reckon is the one on /dev/sda, the disk I'm booted up on.

I just stumbled upon this thread https://askubuntu.com/questions/766048/ … vm2-member and this blog post http://pissedoffadmins.com/os/mount-unk … ember.html, where it seems as if my problem with mounting might arise from the fact that my current system on /dev/sda2 has a VGname of "MyStorage", which is identical to the name used by the partition I'm trying to recover. The first thread advises that volume group names be unique to avoid this error. I guess my question is, what would be the best way to do this, ideally without making any writes to the functional partition, since I don't want to mess another partition up.

Last edited by archfluke (2017-12-16 16:01:07)

frostschutz · 2017-12-16 16:02:46

You can't have two VGs with same name active simultaneously, you have to vgrename one.

vgdisplay to see names and uuids, then it's vgrename vguuid newname or some such

of course if it's readonly you can't rename it... you could use overlays to add a read-write layer on top https://raid.wiki.kernel.org/index.php/ … erlay_file

or just ditch the readonlyness if you're very sure it's the correct thing to do. better to stick to readonly until you have your data safe and sound

Last edited by frostschutz (2017-12-16 16:04:12)

archfluke · 2017-12-16 16:06:20

vgdisplay outputs this:

  --- Volume group ---
  VG Name               MyStorage
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <232.38 GiB
  PE Size               4.00 MiB
  Total PE              59489
  Alloc PE / Size       59489 / <232.38 GiB
  Free  PE / Size       0 / 0   
  VG UUID               <redacted>

Only the volume group for the working partition is showing up. Does that mean that I will have to rename that one? I was also considering just booting into the arch live usb and trying to run the commands there to avoid getting my working partition involved.

archfluke · 2017-12-16 16:33:03

I just rebooted into the live usb and ran vgdisplay which outputs the following:

WARNING: Device /dev/mapper/luksRecover has size of 240789502 sectors which is smaller than corresponding PV size of 250045071 sectors.  Was device resized?  One or more devices used as PVs in VG MyVol have changed sizes.
  --- Volume group ---
  VG Name               MyVol
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               119.23 GiB
  PE Size               4.00 MiB
  Total PE              30522
  Alloc PE / Size       30522 / 119.23 GiB
  Free  PE / Size       0 / 0   
  VG UUID               <redacted>

And

vgchange -ay

outputs this:

WARNING: Device /dev/mapper/luksRecover has size of 240789502 sectors which is smaller than corresponding PV size of 250045071 sectors.  Was device resized?  One or more devices used as PVs in VG MyVol have changed sizes.
device-mapper: resume ioctl on (254:1) failed: Invalid argument
Unable to resume MyVol-root (254:1)
1 logical volume(s) in volume group "MyVol" now active

unfortunately, I can't mount /dev/MyVol as it doesn't even show up in /dev as I expected it to

Last edited by archfluke (2017-12-16 16:33:53)

archfluke · 2017-12-16 16:59:51

I ran losetup without the --read-only flag and just tried to run mount /dev/mapper/luksRecover /mnt and I get this:

[1141.563142] Buffer I/O error on dev sdb4, logical block 244889344, async page read
[1141.563142] Buffer I/O error on dev sdb4, logical block 244889344, async page read
[1141.563142] Buffer I/O error on dev sdb4, logical block 244889344, async page read
[1141.563142] Buffer I/O error on dev sdb4, logical block 244889344, async page read
[1141.563142] Buffer I/O error on dev sdb4, logical block 244889344, async page read
[1141.563142] blk_update_request: I/O error, dev loop1, sector 240793344
[1141.563142] blk_update_request: I/O error, dev loop1, sector 240793344
[1141.563142] blk_update_request: I/O error, dev loop1, sector 240793344
[1141.563142] blk_update_request: I/O error, dev loop1, sector 240793344
[1141.563142] blk_update_request: I/O error, dev loop1, sector 4096
[1141.563142] FAT-fs (dm-0): unable to read boot sector
mount: /dev/mapper/luksRecover: can't read superblock

the block number range is from 244889344 to 244889351 for the async page read errors
the sector number range is from 240793344 to 240793350. I had to manually type the code above and put up the general gist without going into the numerical output wihtin that range, but can refactor if needed for troubleshooting.

Last edited by archfluke (2017-12-16 17:03:08)

frostschutz · 2017-12-16 17:03:38

regarding the too small size, make a loop device not on sdb4 but sdb and add the current offset of sdb4 to the offset.

Or might as well show your partition table (parted /dev/sdb unit s print free) and just adapt the partition table itself...

archfluke · 2017-12-16 17:12:02

if I understand correctly, would I run losetup --find --show --read-only --offset $((0x7d000200)) /dev/sdb and then try cryptsetup opening, and then mounting and other stuff then?

Doing that, I find by the time I get to crypsetup open on /dev/loop0 this error message:

Device /dev/loop0 doesn't exist or access denied.

And then running parted /dev/sdb unit s print free output this:

Model: WD easystore 25FA (scsi)
Disk /dev/sdb: 7813969920s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start       End          Size         File system  Name  Flags
        34s         2047s        2014s        Free Space
 1      2048s       804863s      802816s      fat32              msftdata
 2      804864s     1083391s     278528s      ntfs               msftdata
 3      1083392s    245972990s   244889599s   ntfs               msftdata
 4      245972991s  490862589s   244889599s   ntfs               msftdata
        490862590s  7813969886s  7323107297s  Free Space

I also remembered upon running vgcreate and reviewing its output that the VG name isn't shared, but is "MyVol" vs the "MyStorage" on the working drive. I don't know why it wasn't registering then.

Last edited by archfluke (2017-12-16 17:18:55)

frostschutz · 2017-12-16 17:18:57

You have to add the sdb4 offset to it, 245972991s (now we know why the offset is so odd, who does partitions at odd sector numbers nowadays)

Total offset seems to be 128035323904 bytes or 250068992 512b sectors or 122104 MiB.

According to your output, the size of the PV should be 250045071 sectors, add 2 MiB for LUKS header, which means the correct partition should go from 250068992s to 500116111s... which is still a very odd number, somehow.

I might be wrong in my maths, please check it yourself.

archfluke · 2017-12-16 19:46:39

I was wondering what I would be able to do with the numbers you gave. Would I be using gdisk to delete the currently messed up partition and instead create a new partition with those sector values? Would there be any harm in just trying to be exact with the partition starting point and be generous with the endpoint? I wasn't able to find any syntax instructions for losetup by using sector values. I reckon that's what I'd be needing to do?

archfluke · 2017-12-16 20:13:42

so I used the numbers you gave me by deleting the bad partition and creating a new one using gdisk with the same partition type (8e00) that I used to setup my LUKS on LVM (or was it LVM on LUKS?) and I was able to mount the volume inside! Thank you! I will report back once I've completed copying and verifying the data I find. I reckon I should expect everything to be in tact. I was a bit looser on the end point just because...

frostschutz · 2017-12-16 21:32:35

Well, only you know if there was another partition afterwards and whether you'd be interested in the data on it. It would be fine if it was a swap partition or whatever.

Glad you got your data.

Last edited by frostschutz (2017-12-16 21:32:52)

jjacky · 2017-12-16 21:39:57

frostschutz wrote:

According to your output, the size of the PV should be 250045071 sectors, add 2 MiB for LUKS header, which means the correct partition should go from 250068992s to 500116111s... which is still a very odd number, somehow.

So I was reading this just to see if I got it right myself, and - though I may be wrong - I think there might be a little mistake here: I don't think it would be 500 116 111s, but 500 114 067s
Seems you added 2048 for the 2 MiB LUKS header, but that should have been 4 (512b sectors), or am I mistaken?

frostschutz · 2017-12-16 21:51:00

Sure, my mistake.

The 2MiB was an assumption in the first place, if the LUKS container is very old it might be the old default of 2056 sectors. The LUKS metadata barely doesn't fit in 1MiB so when everything changed to MiB alignment, it was cranked up to 2MiB.

No idea what it's like for LUKS2 - I won't be using that for another 5 years at least.

Last edited by frostschutz (2017-12-16 21:52:17)

archfluke · 2017-12-16 21:54:09

so when I tried to run a simple cp/copy command, I get this message:

cp: error reading 'mnt/home/user/.local/share/heroku/client/node_modules/heroku-ci/test/mocha.opts': Input/output error

I was sure that my journey was at an end as well

would running fsck be advised?

Last edited by archfluke (2017-12-16 22:00:33)

frostschutz · 2017-12-16 22:03:01

dmesg?

smartctl -a /dev/sdb?

If the drive itself is suspect then you might need to ddrescue to a new one. But usually that comes with different messages than what you showed above. Did you shorten or grep the output?

absolutely not fsck if there are hardware or connection problems.

Last edited by frostschutz (2017-12-16 22:04:10)

archfluke · 2017-12-16 22:13:46

ok, no fsck it is then. I just rebooted my computer and tried again and cp -a is working just fine right now. I'm taking a "women and children first approach first" although I hope I can salvage the entire system so I don't have to reinstall everything, since I don't remember which packages I installed and how I configured my dot files, etc.

Would you advise that I stop copying and proceed with the above advised steps and then move forward?

Last edited by archfluke (2017-12-16 22:14:43)

archfluke · 2017-12-17 19:54:00

I built up the pluck to work on the original disk and deleted the bad partition and used gdisk to create a new partition the way I did on the clone and I was able to boot into my arch install and I'm in! Incidentally, I was able to rescue my previously non-bootable Windows partition by changing my ESP, which somehow was set as a MS partition (something that testdisk seems to love to do), to EF00.

Thanks everyone. A special thank you to frostschutz for being super helpful in that initial ubuntu forum thread that I linked to above and for sticking with someone who couldn't tell a mahogany table from a partition table prior to this thread, and led me through with just the right degree of complexity to get the job done! I'll set this to solved now!

Arch Linux

#1 2017-12-16 03:25:46

[SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#2 2017-12-16 03:35:09

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#3 2017-12-16 03:54:42

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#4 2017-12-16 13:51:00

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#5 2017-12-16 15:41:09

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#6 2017-12-16 15:46:59

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#7 2017-12-16 15:54:02

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#8 2017-12-16 15:59:42

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#9 2017-12-16 16:02:46

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#10 2017-12-16 16:06:20

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#11 2017-12-16 16:33:03

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#12 2017-12-16 16:59:51

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#13 2017-12-16 17:03:38

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#14 2017-12-16 17:12:02

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#15 2017-12-16 17:18:57

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#16 2017-12-16 19:46:39

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#17 2017-12-16 20:13:42

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#18 2017-12-16 21:32:35

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#19 2017-12-16 21:39:57

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#20 2017-12-16 21:51:00

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#21 2017-12-16 21:54:09

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#22 2017-12-16 22:03:01

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#23 2017-12-16 22:13:46

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

#24 2017-12-17 19:54:00

Re: [SOLVED] how to recover lost LUKS partition; testdisk rewrote p table

Board footer