You are not logged in.

#1 2019-11-07 20:16:58

whomen
Member
Registered: 2018-02-05
Posts: 7

[solved] restore fs: lvm on luks - /home

hi,

i've been a bit to fast in attempt to shrink my home and enlarge my root partition.

lvreduce -L -10G /mapper/home

so my problem now is that the file system thinks it is 10G bigger than the volume.
i tried to restore my /mapper/home on a dd disk clone: 

resize2fs

didn't let me without running e2fsck before

e2fsck -b <backupsuperblocks1..n>

didn't like the superblocks promted by: mke2fs -n

vgcfgrestore

the /etc/lvm/archive on the /root partition

mke2fs -S

nothing worked..
now i thought i might just format the /home and copy the files back from my backup.
so i did:

mkfs.ext4 /dev/mapper/home

/root and /home seem to be clear.
if i try to boot from the harddisk (with the modified dd copy)  the bootsplash tells me

/dev/mapper/root clean, 423512/983040 files, 3575153/3932150 blocks
[FAILED] Failed to start File System Check on /dev/disk/by-uuid/<uuid>.
[DEPEND] dependency failed for /home
[DEPEND] dependency failed for Local File Systems. 
... 
You are in emergency mode ... 

and i can't do anything except for hitting control-d.
this is by the way the same error that i have when trying to boot with my lvreduce stupidity.

here are some facts:

root@archiso # lsblk
sdb                 8:16   0 931.5G  0 disk  
├─sdb1              8:17   0   550M  0 part  
└─sdb2              8:18   0 476.4G  0 part  
  └─cryptdisk     254:0    0 476.4G  0 crypt 
    ├─mapper-swap 254:1    0     8G  0 lvm   
    ├─mapper-root 254:2    0    25G  0 lvm   
    └─mapper-home 254:3    0 443.4G  0 lvm
root@archiso # pvscan

[J  --- Physical volume ---
  PV Name               /dev/mapper/cryptdisk
  VG Name               mapper
  PV Size               <476.40 GiB / not usable <1.32 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              121958
  Free PE               0
  Allocated PE          121958
  PV UUID               C9T02C-IeKC-7pFs-FRLv-A7vM-Jpw3-gab8P3
root@archiso # lvscan
  --- Logical volume ---
  LV Path                /dev/mapper/swap
  LV Name                swap
  VG Name                mapper
  LV UUID                0e5hZ6-tGOP-umil-6FYS-LdLR-TlSf-1NUZWJ
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-02-05 20:50:52 +0000
  LV Status              available
  # open                 0
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1
   
  --- Logical volume ---
  LV Path                /dev/mapper/root
  LV Name                root
  VG Name                mapper
  LV UUID                cxK2f9-5ifF-kfVr-p3Ck-Ty9d-T5NC-rOJ5X7
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-02-05 20:51:19 +0000
  LV Status              available
  # open                 0
  LV Size                25.00 GiB
  Current LE             6400
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:2
   
  --- Logical volume ---
  LV Path                /dev/mapper/home
  LV Name                home
  VG Name                mapper
  LV UUID                VX9zOn-efob-2TE9-owlA-chy6-t1pE-F2vuRq
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-02-05 20:52:03 +0000
  LV Status              available
  # open                 0
  LV Size                <443.40 GiB
  Current LE             113510
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:3

any suggestions how i could get my /home back?
thanks!

Last edited by whomen (2019-11-08 21:31:38)

Offline

#2 2019-11-07 20:27:51

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [solved] restore fs: lvm on luks - /home

it's good if you have a backup

if this LVM is on SSD, and cryptdisk has allowdiscards, and you have issue_discards=1 in your lvm.conf then lvreduce would have deleted (discard/TRIM) those 10G right away, that damage is permanent and cannot be recovered

otherwise lvextend (if LV was unfragmented and no other LV created since reduce) or restoring from LVM config backup should have helped

mkfs.ext4 also discards the entire partition so all data is gone after mkfs (if this is on SSD etc. as above)

mkfs gives a new random UUID which you have to update in your /etc/fstab. otherwise tune2fs -U $olduuid to set it to the old UUID.

for login as $user to work you have to re-create the homedir too, possibly from /etc/skel with proper owner and permissions

Last edited by frostschutz (2019-11-07 20:31:17)

Offline

#3 2019-11-07 20:57:09

whomen
Member
Registered: 2018-02-05
Posts: 7

Re: [solved] restore fs: lvm on luks - /home

thanks for the quick reply!
i gathered the /home uuid

blkid -o list

and changed it in the fstab but still the same error when booting..

or did i look for the wrong uuid?

this is the backup vg:

mapper {
	id = "iGHLq3-EVLU-bcYm-tVfp-sSBv-pyUM-CfnwAz"
	seqno = 4
	format = "lvm2"			# informational
	status = ["RESIZEABLE", "READ", "WRITE"]
	flags = []
	extent_size = 8192		# 4 Megabytes
	max_lv = 0
	max_pv = 0
	metadata_copies = 0

	physical_volumes {

		pv0 {
			id = "C9T02C-IeKC-7pFs-FRLv-A7vM-Jpw3-gab8P3"
			device = "/dev/mapper/mapper"	# Hint only

			status = ["ALLOCATABLE"]
			flags = []
			dev_size = 999082639	# 476.4 Gigabytes
			pe_start = 2048
			pe_count = 121958	# 476.398 Gigabytes
		}
	}

	logical_volumes {

		swap {
			id = "0e5hZ6-tGOP-umil-6FYS-LdLR-TlSf-1NUZWJ"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			creation_time = 1517863852	# 2018-02-05 21:50:52 +0100
			creation_host = "archiso"
			segment_count = 1

			segment1 {
				start_extent = 0
				extent_count = 2048	# 8 Gigabytes

				type = "striped"
				stripe_count = 1	# linear

				stripes = [
					"pv0", 0
				]
			}
		}

		root {
			id = "cxK2f9-5ifF-kfVr-p3Ck-Ty9d-T5NC-rOJ5X7"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			creation_time = 1517863879	# 2018-02-05 21:51:19 +0100
			creation_host = "archiso"
			segment_count = 1

			segment1 {
				start_extent = 0
				extent_count = 3840	# 15 Gigabytes

				type = "striped"
				stripe_count = 1	# linear

				stripes = [
					"pv0", 2048
				]
			}
		}

		home {
			id = "VX9zOn-efob-2TE9-owlA-chy6-t1pE-F2vuRq"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			creation_time = 1517863923	# 2018-02-05 21:52:03 +0100
			creation_host = "archiso"
			segment_count = 1

			segment1 {
				start_extent = 0
				extent_count = 116070	# 453.398 Gigabytes

				type = "striped"
				stripe_count = 1	# linear

				stripes = [
					"pv0", 5888
				]
			}
		}
	}

}

this is the lvscan output now

  --- Logical volume ---
  LV Path                /dev/mapper/swap
  LV Name                swap
  VG Name                mapper
  LV UUID                0e5hZ6-tGOP-umil-6FYS-LdLR-TlSf-1NUZWJ
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-02-05 20:50:52 +0000
  LV Status              available
  # open                 0
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1
   
  --- Logical volume ---
  LV Path                /dev/mapper/root
  LV Name                root
  VG Name                mapper
  LV UUID                cxK2f9-5ifF-kfVr-p3Ck-Ty9d-T5NC-rOJ5X7
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-02-05 20:51:19 +0000
  LV Status              available
  # open                 1
  LV Size                25.00 GiB
  Current LE             6400
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:2
   
  --- Logical volume ---
  LV Path                /dev/mapper/home
  LV Name                home
  VG Name                mapper
  LV UUID                VX9zOn-efob-2TE9-owlA-chy6-t1pE-F2vuRq
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-02-05 20:52:03 +0000
  LV Status              available
  # open                 0
  LV Size                <443.40 GiB
  Current LE             113510
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:3

the error after the [failed]

see 'systemctl status "systemd-fsck@dev-disk-by\\x2duuid-90cd1836... blabla

probably i have to tell systemd the new uuid as well?

---
EDIT:
now i tried to tune2fs /home with the old uuid. still the same error..

Last edited by whomen (2019-11-07 21:28:03)

Offline

#4 2019-11-07 21:51:58

whomen
Member
Registered: 2018-02-05
Posts: 7

Re: [solved] restore fs: lvm on luks - /home

manual fsck works:

fsck from util-linux 2.31
e2fsck 1.43.7 (16-Oct-2017)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/mapper-home: 11/29065216 files (0.0% non-contiguous), 2105281/116234240 blocks

fdisk -l shows the internal ssd

[1mDisk /dev/nvme0n1: 477 GiB, 512110190592 bytes, 1000215216 sectors
[0mUnits: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E0C3ADC8-67C7-4CB7-ABBF-F47392A64797

[1mDevice[0m         [1m  Start[0m [1m       End[0m [1m  Sectors[0m [1m  Size[0m [1mType[0m
/dev/nvme0n1p1    2048    1128447   1126400   550M EFI System
/dev/nvme0n1p2 1128448 1000215182 999086735 476.4G Linux LVM

and the external disk with the modified backup:

GPT PMBR size mismatch (1000215215 != 1953458175) will be corrected by w(rite). // could this be the issue?
Disk /dev/sdb: 931.5 GiB, 1000170586112 bytes, 1953458176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E0C3ADC8-67C7-4CB7-ABBF-F47392A64797

Device        Start       End          Sectors    Size  Type
/dev/sdb1     2048    1128447   1126400   550M EFI System
/dev/sdb2  1128448 1000215182 999086735 476.4G Linux LVM

Disk /dev/loop0: 408.5 MiB, 428281856 bytes, 836488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/cryptdisk: 476.4 GiB, 511530311168 bytes, 999082639 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/mapper-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/mapper-root: 25 GiB, 26843545600 bytes, 52428800 sectors
[0mUnits: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/mapper-home: 443.4 GiB, 476095447040 bytes, 929873920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Offline

#5 2019-11-07 22:44:10

whomen
Member
Registered: 2018-02-05
Posts: 7

Re: [solved] restore fs: lvm on luks - /home

frostschutz wrote:

it's good if you have a backup

if this LVM is on SSD, and cryptdisk has allowdiscards, and you have issue_discards=1 in your lvm.conf then lvreduce would have deleted (discard/TRIM) those 10G right away, that damage is permanent and cannot be recovered

otherwise lvextend (if LV was unfragmented and no other LV created since reduce) or restoring from LVM config backup should have helped

it's a ssd but i didn't actively activate 'allowdiscards'
the /home still had 70G free space. [btw: with 'testdisk' i was able to backup all content of /home]
since i still have a backup maybe i'll give that a try.

i would then need to reduce the /root again (-10G)

lvreduce --resizefs -L -10G /dev/mapper/root

and then

lvresize -L +10G /dev/mapper/home

right?

Last edited by whomen (2019-11-07 22:46:28)

Offline

#6 2019-11-08 21:04:25

whomen
Member
Registered: 2018-02-05
Posts: 7

Re: [solved] restore fs: lvm on luks - /home

YYAAAH!
systemd-boot.. and boot loader entries..
the problem was that the corrupted disk is always connected.
even when i tried to boot from the repaired usb-backupdisk systemd tried to mount the bad internal ssd.

but still it's not completely solved.. i created the user folder on /home and copied the /skel things but still i'm not able to log in.
maybe because neither x nor i3 or anything can find any configfiles..

but before i'll maybe give the lvresize a shot. then i could forget about all the backup digging..

Offline

#7 2019-11-08 21:30:39

whomen
Member
Registered: 2018-02-05
Posts: 7

Re: [solved] restore fs: lvm on luks - /home

fuck..
i can't belief this. 2 nights no sleep and sweating like shit..

in the end i was brave and tried to just revert what i did:

# resize2fs -p /dev/mapper/root 15G
# e2fsck -f /dev/mapper/root
# lvreduce -L -10G /dev/mapper/root
# lvresize -L +119G /dev/mapper/home

and it just innocently booted if as nothing ever happend!!
at least i've learned something about lvm and systemd.
thanks for your help!

Offline

Board footer

Powered by FluxBB