Arch Linux

espritlibre

Member

Registered: 2022-12-15

Posts: 132

[Solved] how to recover lvm2 volume/partition?

so well... it had to happen at some point...
by accident i nuked a 1TB lvm/ext4 partition, which resides on encrypted luks, with the following command:

sudo dd if=/dev/nvme0n1p1 of=/dev/VolGroup/files bs=64K conv=noerror,sync status=progress

this is my first setup with lvm2 and i have little experience with and gparted doesn't support lvm2, what is the prefered way to recover the volume and its contents? i haven't mounted the partition since...

output of blkid

sudo blkid
/dev/nvme0n1p1: UUID="138E-D91B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="8c0ad4ca-0ca5-47ab-a81c-f3b5644c95a4"
/dev/nvme0n1p2: UUID="4f1ca192-c9f6-4d00-8ce7-a9615f3b1635" TYPE="crypto_LUKS" PARTLABEL="Linux LUKS" PARTUUID="22292a7e-b349-48db-94e1-76e4db685864"
/dev/mapper/cryptlvm: UUID="BifA6o-UugM-Asc8-mrJE-BJEQ-Fu5X-dZuGwf" TYPE="LVM2_member"
/dev/mapper/VolGroup-root: UUID="60d51440-8fd1-4979-a5b5-893601df59bd" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/VolGroup-home: UUID="d51de7b7-07ed-4c47-a6b8-999250409d67" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/VolGroup-files: UUID="138E-D91B" BLOCK_SIZE="512" TYPE="vfat"

any help is appreciated

EDIT:

sudo lvs                           
  LV      VG                   Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  files   VolGroup         -wi-a----- 1023,75g                                                    
  home VolGroup         -wi-ao----  400,00g                                                    
  root   VolGroup         -wi-ao----   64,00g

sudo vgs
  VG                   #PV #LV #SN Attr   VSize  VFree   
  VolGroup              1   3   0 wz--n- <1,60t <150,25g

sudo pvs
  PV                   VG                        Fmt  Attr PSize    PFree   
  /dev/mapper/cryptlvm VolGroup lvm2 a--  <1,60t <150,25g

sudo lsblk               
NAME                             MAJ:MIN  RM    SIZE RO TYPE  MOUNTPOINTS
nvme0n1                          259:0    0    1,8T  0 disk  
├─nvme0n1p1                      259:1    0    512M  0 part   /efi
└─nvme0n1p2                      259:2    0    1,6T   0 part  
  └─cryptlvm                         254:0    0    1,6T   0 crypt 
    ├─VolGroup-root  254:1                 0     64G   0 lvm    /
    ├─VolGroup-home  254:2               0    400G  0 lvm   /home
    └─VolGroup-files 254:3                  0 1023,8G  0 lvm

sudo vgcfgrestore --list VolGroup
   
  File:		/etc/lvm/archive/VolGroup_00000-1094159472.vg/VolGroup_00000-1094159472.vg
  VG name:    	VolGroup
  Description:	Created *before* executing 'vgscan'
  Backup Time:	Sun Oct  9 18:09:29 2022

   
  File:		/etc/lvm/archive/VolGroup_00001-285462029.vg/VolGroup_00001-285462029.vg
  VG name:    	VolGroup
  Description:	Created *before* executing 'lvreduce -L -256M VolGroup/files'
  Backup Time:	Sun May 14 14:45:58 2023

   
  File:		/etc/lvm/backup/VolGroup/VolGroup
  VG name:    	VolGroup
  Description:	Created *after* executing 'lvreduce -L -256M VolGroup/files'
  Backup Time:	Sun May 14 14:45:58 2023

since the last backup  i haven't made changes to the filesystem but i'm unsure if i should just yolo and execute...

sudo vgcfgrestore VolGroup

... or not? atm i don't have a drive where i could backup 1TB of data in case this goes wrong and i loose all contents of the volume...

Last edited by espritlibre (2023-11-11 22:21:38)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,474

Re: [Solved] how to recover lvm2 volume/partition?

Well, still running or not, it's already overwritten. How large is nvme0n1p1? I assume it was copied fully. That's kind of deadly to any filesystem...

I would zero out the area that has been overwritten (no data is left there, and it will only confuse recovery tools), then try your luck with fsck/debugfs, backup superblocks, or photorec and others.

You can do the whole thing on a snapshot / copy-on-write or make a full image for experiments if you have the space for it somewhere.

PS: noerror,sync will do nothing at best and make your copy useless/corrupt if it actually encounters any error. best not to use it at all, if you expect errors go with ddrescue instead. if you must use it anyway, also add iflag=fullblock.

Last edited by frostschutz (2023-10-28 21:20:30)

espritlibre

Member

Registered: 2022-12-15

Posts: 132

Re: [Solved] how to recover lvm2 volume/partition?

thanks for the help frostschutz, i edited my first post just to see that you ninjaed me.

EDIT:
nvme0n1p1 is my ESP, it is 537 MB, the nuked volume is 1TB and about half full... in the best scenario it didn't overwrite anything

Last edited by espritlibre (2023-10-28 21:37:29)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,474

Re: [Solved] how to recover lvm2 volume/partition?

vgcfgrestore won't help you at all. You didn't damage LVM metadata. You wiped the first 512M of your logical volume / filesystem. You'd have the same issue with regular partition (/dev/sdx9) instead of LVM.

espritlibre

Member

Registered: 2022-12-15

Posts: 132

Re: [Solved] how to recover lvm2 volume/partition?

I would zero out the area that has been overwritten (no data is left there, and it will only confuse recovery tools),

how would i do that correctly, would:

dd if=/dev/zero of=/dev/VolGroup/files bs=563085312 count=1

do? (537MB=563085312bytes)

unfortunately my backup drives are all full, so monday i'm gonna buy another ssd and then attempt to recover my files.
thanks so far for your guidance frostschutz, i'll report back...

Last edited by espritlibre (2023-10-29 00:08:13)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,474

Re: [Solved] how to recover lvm2 volume/partition?

dd with large blocksize is unreliable, add iflag=fullblock to make sure it actually does use one full block.

Or just use `head -c 12345 /dev/zero > /dev/VG/LV`. You can get the size in bytes with `blockdev --getsize64 /dev/nvme0n1p1`.

Before that, you can use `cmp -l /dev/nvme0n1p1 /dev/VolGroup/files` to verify it has been overwritten fully. cmp will print differences; if there aren't any, it's identical. If it was not fully overwritten it should print lots of differences for larger byte offsets.

cmp won't work well if either partition has been modified since.

I still recommend you do all experiments on a snapshot or copy. Consider `lvchange --permission r VolGroup/files` to make that LV readonly so no more mischief can happen to it in the meantime.

----

Recent dd also supports `count=500MiB` (unit MUST end with B, so 500M does NOT work - it's very confusing but that's dd for you).

$ dd if=/dev/zero count=500MiB | wc --bytes
1024000+0 records in
1024000+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.427136 s, 1.2 GB/s524288000
$ dd if=/dev/zero count=500M | wc --bytes
[... killall -USR1 dd ...]
103117378+0 records in
103117378+0 records out
52796097536 bytes (53 GB, 49 GiB) copied, 39.57 s, 1.3 GB/s
[... keeps going ...]

Using dd for this is dangerous, it's dangerous even if you know dd well, since dd suddenly introduces new behavior like this

Last edited by frostschutz (2023-10-29 07:29:30)

espritlibre

Member

Registered: 2022-12-15

Posts: 132

Re: [Solved] how to recover lvm2 volume/partition?

ftr
i followed the ddrescue wiki --> created the image, then fscked the partition. fsck complained about missing the superblock so i gave fsck the rescue superblocks to work with. i had noted down the superblocks, when i created the partition... unfortunately fsck wasn't able to recover the files, i tried all rescue superblocks which probably made the situation even worse. after my unsuccessful attempt to recover the partition, i gave up on it and took the chance to change the partition layout. i still have the image but it's probably useless because it resides on a usb attached ssd.

luckily i lost no data because i have backups in place, my initial panic was groundless.

sorry for writing from memory and getting back to this topic late, amazon did delay the delivery of the new drive + i was busy myself.