Arch Linux

traches

Member

Registered: 2019-03-27

Posts: 12

[SOLVED] Can't activate LVM volume group after power failure

Hello! sorry to come back again with another problem, and thanks for your time. I'm pretty sure this one is less dumb than the last one. (Don't look, I was really dumb.)

I have 3 drives set up as LVM physical volumes, with one volume group containing a raid 5 logical volume which I use to store large files (not the root filesystem):

[root@charybdis /]# pvs
  PV         VG             Fmt  Attr PSize  PFree
  /dev/sda1  volume_group_1 lvm2 a--  <1.82t    0 
  /dev/sdb1  volume_group_1 lvm2 a--  <1.82t    0 
  /dev/sdc1  volume_group_1 lvm2 a--  <1.82t    0 
[root@charybdis /]# vgs
  VG             #PV #LV #SN Attr   VSize  VFree
  volume_group_1   3   1   0 wz--n- <5.46t    0 
[root@charybdis /]# lvs
  LV            VG             Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  chuckleBucket volume_group_1 rwi---r--- <3.64t                    

I had a power failure, and am now unable to mount it. The volume group doesn't show up under /dev/mapper:

[root@charybdis /]# cd /dev/mapper
[root@charybdis mapper]# ls
control  home

and I get the following error when I try to activate it:

[root@charybdis mapper]# vgchange -ay
  device-mapper: reload ioctl on  (254:7) failed: Input/output error
  0 logical volume(s) in volume group "volume_group_1" now active
[root@charybdis mapper]# ls
control  volume_group_1-chuckleBucket_rimage_0	volume_group_1-chuckleBucket_rimage_2  volume_group_1-chuckleBucket_rmeta_1
home	volume_group_1-chuckleBucket_rimage_1	volume_group_1-chuckleBucket_rmeta_0   volume_group_1-chuckleBucket_rmeta_2
[root@charybdis mapper]# vgchange -ay
  Activation of logical volume volume_group_1/chuckleBucket is prohibited while logical volume volume_group_1/chuckleBucket_rimage_0 is active.
  0 logical volume(s) in volume group "volume_group_1" now active
[root@charybdis mapper]# ls
control  volume_group_1-chuckleBucket_rimage_0	volume_group_1-chuckleBucket_rimage_2  volume_group_1-chuckleBucket_rmeta_1
home	volume_group_1-chuckleBucket_rimage_1	volume_group_1-chuckleBucket_rmeta_0   volume_group_1-chuckleBucket_rmeta_2
[root@charybdis mapper]# vgchange -an
  0 logical volume(s) in volume group "volume_group_1" now active
[root@charybdis mapper]# ls
control  home
[root@charybdis mapper]# 

Obviously something is happening, since the 'rimage' and 'rmeta' volumes show up, but I don't know what to do with them. I tried a refresh:

vgchange --refresh

But it has no output, and apparently no effect. I've tried restoring a backup configuration from /etc/lvm/backup/, but I still get the same errors.

DuckDuckGo turns up nothing regarding this particular problem, and I've dug pretty far into the LVM man pages without any illumination. I do have backups, but it's ~2 terabytes of data so recovering would be far more convenient than restoring.

I don't need handholding, but could really use a little direction. Thank you!

Last edited by traches (2019-10-08 13:03:03)

Swiggles

Member

Registered: 2014-08-02

Posts: 266

Re: [SOLVED] Can't activate LVM volume group after power failure

First of all let's see what lvm has to say about it:

pvscan
vgscan
lvscan
vgcfgrestore --list volume_group_1

Depending on what the error actually is the likely tools to check are

lvconvert --repair ...
vgcfgrestore ...

Hello! sorry to come back again with another problem, and thanks for your time. I'm pretty sure this one is less dumb than the last one. (Don't look, I was really dumb.)

You know I had to check? Could be worse. haha

traches

Member

Registered: 2019-03-27

Posts: 12

Re: [SOLVED] Can't activate LVM volume group after power failure

[root@charybdis mapper]# pvscan
  PV /dev/sda1   VG volume_group_1   lvm2 [<1.82 TiB / 0    free]
  PV /dev/sdb1   VG volume_group_1   lvm2 [<1.82 TiB / 0    free]
  PV /dev/sdc1   VG volume_group_1   lvm2 [<1.82 TiB / 0    free]
  Total: 3 [<5.46 TiB] / in use: 3 [<5.46 TiB] / in no VG: 0 [0   ]
[root@charybdis mapper]# vgscan
  Reading volume groups from cache.
  Found volume group "volume_group_1" using metadata type lvm2
[root@charybdis mapper]# lvscan
  inactive          '/dev/volume_group_1/chuckleBucket' [<3.64 TiB] inherit
[root@charybdis mapper]# vgcfgrestore --list volume_group_1
   
  File:		/etc/lvm/archive/volume_group_1_00000-314631764.vg
  Couldn't find device with uuid YQcJGn-gzz6-6001-gNxn-nU8X-m5Hp-ZrN51n.
  Couldn't find device with uuid ILQEDc-iuKE-u3Mj-gf9b-6gCQ-cuIl-su8dGo.
  Couldn't find device with uuid gE00ch-ZV5K-qYCw-kjAo-yhT1-luok-nN0dnm.
  VG name:    	volume_group_1
  Description:	Created *before* executing 'pvscan --cache --activate ay 8:17'
  Backup Time:	Thu Feb 21 18:10:35 2019

   
  File:		/etc/lvm/archive/volume_group_1_00001-1423516865.vg
  VG name:    	volume_group_1
  Description:	Created *before* executing 'lvresize -l +100%FREE --resizefs /dev/volume_group_1/chuckleBucket'
  Backup Time:	Sun May 12 17:57:38 2019

   
  File:		/etc/lvm/archive/volume_group_1_00002-1091732253.vg
  VG name:    	volume_group_1
  Description:	Created *before* executing 'lvresize -l +99%FREE --resizefs /dev/volume_group_1/chuckleBucket'
  Backup Time:	Sun May 12 17:58:05 2019

   
  File:		/etc/lvm/archive/volume_group_1_00003-657276657.vg
  VG name:    	volume_group_1
  Description:	Created *before* executing 'vgscan'
  Backup Time:	Sun Oct  6 18:27:31 2019

   
  File:		/etc/lvm/archive/volume_group_1_00004-1467608234.vg
  VG name:    	volume_group_1
  Description:	Created *before* executing 'vgscan'
  Backup Time:	Sun Oct  6 18:27:31 2019

   
  File:		/etc/lvm/backup/volume_group_1
  VG name:    	volume_group_1
  Description:	Created *after* executing 'vgscan'
  Backup Time:	Sun Oct  6 18:27:31 2019

So it looks like all the parts are getting detected. I tried backing up the current config and restoring the May 12th config, which generated the same error (so I reverted to the most recent configuration). vgcfgrestore doesn't work, because it wants the volume group to be activated:

[root@charybdis mapper]# lvconvert --repair volume_group_1/chuckleBucket
  WARNING: Not using lvmetad because of repair.
  volume_group_1/chuckleBucket must be active to perform this operation.

You know I had to check? Could be worse. haha

Definitely one of my better headdesk moments... At least it's good for a chuckle. Keeps my sense of humility healthy.

I really appreciate your help!

Swiggles

Member

Registered: 2014-08-02

Posts: 266

Re: [SOLVED] Can't activate LVM volume group after power failure

Does partial or degraded activation work in this case?
Check the man page of vgchange (-P I think). Sorry I am currently not on my machine, but the option should be listed there.

Edit: degraded and repair should be the proper action here.

Last edited by Swiggles (2019-10-06 23:07:57)

traches

Member

Registered: 2019-03-27

Posts: 12

Re: [SOLVED] Can't activate LVM volume group after power failure

Doesn't look like it:

[root@charybdis mapper]# vgchange -ay --activation-mode degraded
  device-mapper: reload ioctl on  (254:7) failed: Input/output error
  0 logical volume(s) in volume group "volume_group_1" now active
[root@charybdis mapper]# ls
control  volume_group_1-chuckleBucket_rimage_0	volume_group_1-chuckleBucket_rimage_2  volume_group_1-chuckleBucket_rmeta_1
home	volume_group_1-chuckleBucket_rimage_1	volume_group_1-chuckleBucket_rmeta_0   volume_group_1-chuckleBucket_rmeta_2
[root@charybdis mapper]# vgchange -an
  0 logical volume(s) in volume group "volume_group_1" now active
[root@charybdis mapper]# ls
control  home
[root@charybdis mapper]# vgchange -ay --activation-mode partial
  PARTIAL MODE. Incomplete logical volumes will be processed.
  device-mapper: reload ioctl on  (254:7) failed: Input/output error
  0 logical volume(s) in volume group "volume_group_1" now active
[root@charybdis mapper]# ls
control  volume_group_1-chuckleBucket_rimage_0	volume_group_1-chuckleBucket_rimage_2  volume_group_1-chuckleBucket_rmeta_1
home	volume_group_1-chuckleBucket_rimage_1	volume_group_1-chuckleBucket_rmeta_0   volume_group_1-chuckleBucket_rmeta_2

Swiggles

Member

Registered: 2014-08-02

Posts: 266

Re: [SOLVED] Can't activate LVM volume group after power failure

What bothers me is that device-mapper is complaining. Maybe take down the whole vg and restart it instead?

#Will most likely fail, because it is already inactive anyway
lvchange -an volume_group_1/chuckleBucket
vgchange -an volume_group_1
vgscan
vgchange -ay volume_group1
lvscan
lvchange -ay volume_group_1/chuckleBucket

You could try removing the drives between reactivation if it does not work:

blockdev --flushbufs /dev/sda /dev/sdb /dev/sdc
echo 1 > /sys/block/sda/device/delete
echo 1 > /sys/block/sdb/device/delete
echo 1 > /sys/block/sdc/device/delete

You might have to reboot after.

traches

Member

Registered: 2019-03-27

Posts: 12

Re: [SOLVED] Can't activate LVM volume group after power failure

Sorry for the delay, life happens.

No dice; I lost the console output because of the reboot, but I ran those commands verbatim and still get exactly the same errors.

You think I should just restore from backup? I'm starting to think that might be the easiest option; it'll probably take a day or two but I'll be up and running again.

Swiggles

Member

Registered: 2014-08-02

Posts: 266

Re: [SOLVED] Can't activate LVM volume group after power failure

Maybe. There is no guarantee it will work after and I don't have a magic bullet, but to try some troubleshooting steps.
So restoring should be the safest option.

traches

Member

Registered: 2019-03-27

Posts: 12

Re: [SOLVED] Can't activate LVM volume group after power failure

I got brave and tried --force, still didn't do it:

[root@charybdis mapper]# vgchange --activate y --force volume_group_1
  device-mapper: reload ioctl on  (254:7) failed: Input/output error
  0 logical volume(s) in volume group "volume_group_1" now active

I think I'll wipe these drives and rebuild the array. I'm thinking this time I'll use mdadm instead of lvmraid; I understand it's a little more robust and has a little more history. Apparently LVM uses mdadm under the hood, but none of its tools could recognize my devices so it didn't help me out much. If this were my only copy of this data I'd put more time into it, but for now I'm just really glad I have my backups squared away. I'm into photography, and this data is basically all the photos I've ever taken over the years.

Hey, I really appreciate your help. You've been awesome and patient and even though we didn't get it figured out, I learned a lot and will hopefully set things up a bit better next time. Thanks so much!