Arch Linux

Awlex

Member

Registered: 2018-08-06

Posts: 17

[FAILED] LVM | Cannot convert linear logical volume to raid1

I want to add a 3rd harddrive to my server and switch from linear to raid5 with lvm. Before I can do that however, lvconvert needs to convert it from linear to raid1. When I try to do that however I'm hit with the following error

$ sudo lvconvert --type raid1  my_vg/my_lv
Are you sure you want to convert linear LV media_storage/anime_vol to raid1 type? [y/n]: y
  device-mapper: reload ioctl on  (253:0) failed: Invalid argument
  Failed to suspend logical volume my_vg/my_lv.

After a couple of logical volumes have been created

  LV                 VG            Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  my_lv          my_vg -wi-ao---- 7.25t
  my_lv_rimage_1 my_vg -wi------- 7.25t
  my_lv_rmeta_0  my_vg -wi------- 4.00m
  my_lv_rmeta_1  my_vg -wi------- 4.00m

A consequtive run results in the following error, likely because of the generated logical volumes

$ sudo lvconvert --type raid1  media_storage/anime_vol
Are you sure you want to convert linear LV my_vg/my_lv to raid1 type? [y/n]: y
  Insufficient free space: 1900545 extents needed, but only 14349 available

I couldn't find anything specifically relating to the first error, I have it unmounted before I start, so I'm not sure why it would need to be suspended in the first place. Any idea what might cause this?

Last edited by Awlex (2025-04-04 22:01:32)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,531

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

Any chance you updated kernel shortly before trying to run this command, and the necessary dm/raid module couldn't be loaded as it was removed by the update? So after prepping the subsequent action failed, and LVM didn't clean it up?

Not sure if this is possible. Invalid argument when ioctl is usually a problem within kernel realm (or buggy software calling ioctl wrong). So ioctl failing because of missing kernel module would be one possibility.

Otherwise it's just "a bug". I still use mdadm for raid, and LVM as a better partitioning tool only. It's easy to hit weird bugs with rarely used features... :-/

Last edited by frostschutz (2025-03-30 18:18:36)

Awlex

Member

Registered: 2018-08-06

Posts: 17

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

I had 6.12.10 installed. Updated to 6.13.8 and still have the same issues unfortunately.

I'll try to create a new volume group with a raid1 volume and copy all the data there, then remove the old one and upgrade to raid5

Awlex

Member

Registered: 2018-08-06

Posts: 17

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

So, I've did the following steps and now I'm kinda stuck again.

• Moved as much data as possible from the volume elsewhere, to fit it on a single disk

• Removed the now empty drive from the volume + volume group

• Created a new volume group + raid1 volume with the other 2 drives

• copied all data from the old volume to the new one

• after confirming that there are no problems with the copy I removed the drive from the old volume + volume group and deleted it

• Added the drive with the data to the new volume and converted tried to convert it to raid5

  $ sudo lvconvert --type raid5 --stripes 2 -I 1M my_new_vg/my_new_lv

Now this is the point where I'm stuck. AFAIK I should now have a 14.5T volume. lvs thinks so as well

$ sudo lvs
  LV        VG            Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  my_new_lv my_new_vg     rwi-a-r--- 14.50t                                    100.00

sudo lvs -a -o name,copy_percent,devices
  LV                    Cpy%Sync Devices
  my_new_vol            100.00   my_new_vol_rimage_0(0),my_new_vol_rimage_1(0),my_new_vol_rimage_2(0)
  [my_new_vol_rimage_0]          /dev/sdb1(1)
  [my_new_vol_rimage_1]          /dev/sdc(1)
  [my_new_vol_rimage_2]          /dev/sda1(1)
  [my_new_vol_rmeta_0]           /dev/sdb1(0)
  [my_new_vol_rmeta_1]           /dev/sdc(0)
  [my_new_vol_rmeta_2]           /dev/sda1(0)

However, the file system is still 7.25 and and I cannot extend it

sudo resize2fs /dev/my_new_vg/my_new_lv
resize2fs 1.47.2 (1-Jan-2025)
The filesystem is already 1946157056 (4k) blocks long.  Nothing to do!

and at this point I'm stuck on what to do. Any idea?

Not sure if this is relevant, but I have not deleted the data on the old volume. When I added the drive to the new volume group it was immediatelly synced

-thc

Member

Registered: 2017-03-15

Posts: 848

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

What's the output of

lvs -a -o lv_name,size,segtype,seg_pe_ranges,reshape_len_le,data_stripes

?

Awlex

Member

Registered: 2018-08-06

Posts: 17

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

This was the output:

$ sudo lvs -a -o lv_name,size,segtype,seg_pe_ranges,reshape_len_le,data_stripes
  WARNING: RaidLV my_vg/my_lv needs to be refreshed!  See character 'r' at position 9 in the RaidLV's attributes and its SubLV(s).
  LV                   LSize  Type   PE Ranges                                                                              RSize #DStr
  my_lv                14.55t raid5  my_lv_rimage_0:0-1907718 my_lv_rimage_1:0-1907718 my_lv_rimage_2:0-1907718             0     2
  [my_lv_rimage_0]     <7.28t linear /dev/sdb1:1-1907719                                                                          1
  [my_lv_rimage_1]     <7.28t linear /dev/sdc:1-1907719                                                                           1
  [my_lv_rimage_2]     <7.28t linear /dev/sda1:1-1907719                                                                          1
  [my_lv_rmeta_0]       4.00m linear /dev/sdb1:0-0                                                                                1
  [my_lv_rmeta_1]       4.00m linear /dev/sdc:0-0                                                                                 1
  [my_lv_rmeta_2]       4.00m linear /dev/sda1:0-0  

Note: I executed the following command beforehand, while searching for a solution, which is why it's slightly larger now (I did not save the output, but it did not show an error)

$ sudo lvextend -l 100%FREE -r my_vg/my_lv 

Following this warning I did the following

$ sudo lvchange --refresh my_vg/my_lv # No effect/output

sudo lvconvert --repair my_vg/my_lv
  Active raid has a wrong number of raid images!
  Metadata says 3, kernel says 2.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  my_vg/my_lv does not contain devices specified to replace.
  Faulty devices in my_vg/my_lv successfully replaced.

On top of that, I there might be an issue with the file system as well that causes this.

$ sudo fsck -f /dev/my_vg/my_lv
fsck from util-linux 2.41
e2fsck 1.47.2 (1-Jan-2025)
fsck.ext2: Input/output error while trying to open /dev/mapper/my_vg-my_lv

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

-thc

Member

Registered: 2017-03-15

Posts: 848

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

Here's my analysis on everything so far:

$ sudo lvconvert --type raid1  my_vg/my_lv
Are you sure you want to convert linear LV media_storage/anime_vol to raid1 type? [y/n]: y
  device-mapper: reload ioctl on  (253:0) failed: Invalid argument
  Failed to suspend logical volume my_vg/my_lv.

These errors (253:0 is the device number for your logical volume) are (to me) equivalent of hardware device errors from normal hard disks. Something's already flaky here.

  LV                 VG            Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  my_lv          my_vg -wi-ao---- 7.25t
  my_lv_rimage_1 my_vg -wi------- 7.25t
  my_lv_rmeta_0  my_vg -wi------- 4.00m
  my_lv_rmeta_1  my_vg -wi------- 4.00m

You can see that "my_lv_rimage_0" is missing - this is not a proper RAID1 volume.

$ sudo lvconvert --type raid1  media_storage/anime_vol
Are you sure you want to convert linear LV my_vg/my_lv to raid1 type? [y/n]: y
  Insufficient free space: 1900545 extents needed, but only 14349 available

This error means that you try to convert a linear LV with a size of 1900545 extents (7.25 TB) to a RAID1 LV  - which needs another 1900545 extents - but only 14349 physical extents are free. This cannot work.

On your second try you did

$ sudo lvconvert --type raid5 --stripes 2 -I 1M my_new_vg/my_new_lv

But it's actually a two step process. First convert it to type raid5 (without "--stripes") to an intermediate state and secondly add another image to reach the final state.

This

$ sudo lvs -a -o lv_name,size,segtype,seg_pe_ranges,reshape_len_le,data_stripes
  WARNING: RaidLV my_vg/my_lv needs to be refreshed!  See character 'r' at position 9 in the RaidLV's attributes and its SubLV(s).
  LV                   LSize  Type   PE Ranges                                                                              RSize #DStr
  my_lv                14.55t raid5  my_lv_rimage_0:0-1907718 my_lv_rimage_1:0-1907718 my_lv_rimage_2:0-1907718             0     2
  [my_lv_rimage_0]     <7.28t linear /dev/sdb1:1-1907719                                                                          1
  [my_lv_rimage_1]     <7.28t linear /dev/sdc:1-1907719                                                                           1
  [my_lv_rimage_2]     <7.28t linear /dev/sda1:1-1907719                                                                          1
  [my_lv_rmeta_0]       4.00m linear /dev/sdb1:0-0                                                                                1
  [my_lv_rmeta_1]       4.00m linear /dev/sdc:0-0                                                                                 1
  [my_lv_rmeta_2]       4.00m linear /dev/sda1:0-0  

shows that the RAID is in a weird state (3 subimages but only 2 stripes, no reshape space).

This

$ sudo lvextend -l 100%FREE -r my_vg/my_lv 

makes no sense to me - a conversion to RAID5 needs at least one free PE per member (reshape space).

This

sudo lvconvert --repair my_vg/my_lv
  Active raid has a wrong number of raid images!
  Metadata says 3, kernel says 2.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  my_vg/my_lv does not contain devices specified to replace.
  Faulty devices in my_vg/my_lv successfully replaced.

confirms the "weird state". It would be interesting to see the output of my "lvs" command after this "repair".

This

$ sudo fsck -f /dev/my_vg/my_lv
fsck from util-linux 2.41
e2fsck 1.47.2 (1-Jan-2025)
fsck.ext2: Input/output error while trying to open /dev/mapper/my_vg-my_lv

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

looks like you severely damaged the file system. Hopefully you have a backup.

Awlex

Member

Registered: 2018-08-06

Posts: 17

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

All the files are still there and the file system and I can still write files within those last few remaining GB.
This drive only contains ephemeral data, but I think I'll buy a new drive add a new 8Tb+ drive and copy all the data there and start anew. This time immediately creating a raid5 setup

Awlex

Member

Registered: 2018-08-06

Posts: 17

Re: [FAILED] LVM | Cannot convert linear logical volume to raid1

Nevermind, it's actually gone. Might be a busy weekend, but no truly important data was lost. Thanks everyone for your assistance and expertise!