Arch Linux

vecino

Member

Registered: 2022-12-10

Posts: 139

[SOLVED] Raid1 - mdadm

Hi, can someone please tell me what I'm doing wrong?

I am trying to simulate working with raid1 via mdadm in VirtualBox. Everything works for me, but the moment I remove the disk from the array (in VirtualBox), it does not boot. I just want to simulate when one of the hdd dies.

ERROR: device 13dc3460-a6b3-4f76-86e7-53f937485d90 not found. Skipping fsck.

If both drives are connected, everything works:

[root@archlinux ~]# cat /etc/fstab 
# /dev/md0p1
UUID=13dc3460-a6b3-4f76-86e7-53f937485d90	/         	ext4      	rw,relatime	0 1

[root@archlinux ~]# cat /etc/mdadm.conf
ARRAY /dev/md0 metadata=1.2 name=archiso:0 UUID=089398a9:2ccd866a:7be3d575:6b7742e4

[root@archlinux ~]# lsblk 
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0     8G  0 disk  
└─sda1        8:1    0     8G  0 part  
  └─md0       9:0    0     8G  0 raid1 
    └─md0p1 259:0    0     8G  0 part  /
sdb           8:16   0     8G  0 disk  
└─sdb1        8:17   0     8G  0 part  
  └─md0       9:0    0     8G  0 raid1 
    └─md0p1 259:0    0     8G  0 part  /
sr0          11:0    1 824.3M  0 rom
   
[root@archlinux ~]# blkid 
/dev/sdb1: UUID="089398a9-2ccd-866a-7be3-d5756b7742e4" UUID_SUB="40156b83-8879-749d-8215-4b76af72ab91" LABEL="archiso:0" TYPE="linux_raid_member" PARTUUID="78fb6dba-01"
/dev/md0p1: UUID="13dc3460-a6b3-4f76-86e7-53f937485d90" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="23dbe843-01"
/dev/sr0: BLOCK_SIZE="2048" UUID="2023-02-01-09-09-31-00" LABEL="ARCH_202302" TYPE="iso9660" PTUUID="bdeca99d" PTTYPE="dos"
/dev/sda1: UUID="089398a9-2ccd-866a-7be3-d5756b7742e4" UUID_SUB="6aa5b5ad-6722-5dca-cd97-06fd834df56b" LABEL="archiso:0" TYPE="linux_raid_member" PARTUUID="97ab9ad3-01"

If I understand correctly, I need the system to create /dev/md0 even with one disk? So that it is not dependent on both - sda1 and sdb2 ?

Thanks

I have read these instructions:
https://wiki.archlinux.org/title/RAID
https://wiki.archlinux.org/title/Instal … _Fake_RAID
https://www.serveradminz.com/blog/insta … ware-raid/

Last edited by vecino (2023-02-27 07:04:11)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: [SOLVED] Raid1 - mdadm

It depends on how the array is assembled early boot. ArchLinux initcpio doesn't have a mdadm runtime hook anymore. Instead it just installs udev rules and leaves everything to udev.

And what udev (/lib/udev/rules.d/64-md-raid-assembly.rules) does is "incremental assembly" i.e. it builds md device up as md member devices appear and are detected by udev and rules run for it. You can read more about incremental assembly in the mdadm manpage.

And if all drives are not there, this assembly is unfinished, stuck and waits for more drives to appear... indefinitely. There is nothing that ends this process or forces incomplete arrays to run before the following hooks actually try to mount it. As such all following steps fail.

If you are dropped to a rescue shell at this point you should see in /proc/mdstat one partially assembled array which is not running and thus not usable yet.

The mdadm manpage suggests running 'mdadm -IRs' to run whatever is available at this point. So if you want to change this behavior, you can put this command in a custom hook, placed directly before any hooks that want to use the raid device (before resume, encrypt, …).

Alternatively you can ditch the mdadm_udev hook / incremental assembly altogether and make your custom mdadm hook that assembles it the old way.

Last edited by frostschutz (2023-02-25 22:21:48)

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - mdadm

Thank you frostschutz for your response. I admit that my knowledge of this is insufficient. I don't know how I can solve this.

Thanks too for the "mdadm -IRs" command ... that builds the array, but I don't know how to proceed further to boot. I would need to automate this because I use that raid1 on a remote router with two SSDs just in case one fails so that it runs. This way it will die just like with one disk (after reboot).

I tried the same with Debian 11 and there it is working out of box. Could I ask some experienced users how to solve this in the simplest way?

Thanks!

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - mdadm

Ah, so I boot using the command:

mdadm -IRs
mount /dev/md0p1 /new_root
exit

Please how to automate it? Without user intervention?

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: [SOLVED] Raid1 - mdadm

Make a custom initcpio hook (look at existing hooks for examples, you can also find other custom hooks in these forums, on github, etc.) that just runs the mdadm -IRs command, that should do it.

See also man mkinitcpio, https://wiki.archlinux.org/title/Mkinit … time_hooks ...

Note that by auto-assembling degraded raids, you no longer notice it's degraded. Some people run RAID in degraded mode for years without noticing. Then when another drive fails, they find the mirror drive does not have any data on it...

So you'll also want to set up monitoring with mail notification etc. (MAILADDR in mdadm.conf and mdadm monitor service) and make sure your mailer actually works.

Good luck,

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - mdadm

I found in the history where someone already solved this error. All is here + workaround: https://bugs.archlinux.org/task/57860

I'll summarize it here (maybe it will help someone):

create a file /etc/initcpio/install/my_raid with:

#!/bin/bash

build() {
    add_checked_modules -f 'dm-' 'drivers/md/*'

    # check if a custom mdadm.conf exists
    if grep -qw ^ARRAY "$BASEDIR/etc/mdadm.conf"; then
        echo "Custom /etc/mdadm.conf file will be used in initramfs for assembling arrays."
        add_file "/etc/mdadm.conf"
    fi

    add_binary "/usr/bin/mdadm"
    add_file "/usr/lib/udev/rules.d/63-md-raid-arrays.rules"
    add_file "/usr/lib/udev/rules.d/64-md-raid-assembly.rules"

    add_runscript
}

create a file /etc/initcpio/hooks/my_raid with:

#!/usr/bin/ash

run_hook() {
    
    local seconds=10

    # quick exit
    mdadm --detail --scan | grep -q INACTIVE-ARRAY || return 0

    # wait for slow block devices
    echo "Waiting $seconds seconds for incomplete mdadm arrays..."
    sleep $seconds

    # try to complete assembling and start arrays, possibly in degraded mode
    mdadm -IRs
}

Finally, you need to add my_raid hook to /etc/mkinitcpio.conf and remove mdadm_udev.

HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block my_raid filesystems fsck)
BINARIES="/sbin/mdmon"

mkinitcpio -p linux
..
 -> Running build hook: [my_raid]

Last edited by vecino (2023-02-26 15:48:23)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: [SOLVED] Raid1 - mdadm

If you patch files in /usr/lib/initcpio, then every update to mkinitcpio will overwrite your changes.

You should instead place it as a custom hook in /etc/initcpio/{hooks,install}/ under a new name so there will be no conflict with the standard mdadm_udev hook.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - mdadm

Oh well - thanks for the heads up. I will edit it as you write.

Edit: I updated my post #6 for finally resolve.

Last edited by vecino (2023-02-26 15:51:48)

tz4678

Member

From: St. Petersburg

Registered: 2019-07-30

Posts: 10

Website

Re: [SOLVED] Raid1 - mdadm

Oh well - thanks for the heads up. I will edit it as you write.

Edit: I updated my post #6 for finally resolve.

I have same issue. You don't need custom hooks.

First modify ur

/etc/mkinitcpio.conf

:

HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole block mdadm_udev sd-encrypt filesystems fsck)

Important: replace udev hook to systemd.

Update initramfs:

sudo mkinitcpio -P

Then modify your systemd boot configs like this:

# /boot/loader/entries/arch.conf
options md=0,/dev/nvme0n1p1,/dev/nvme1n1p1 rd.luks.name=</dev/md0 UUID>=root root=/dev/mapper/root rootflags=rw,subvol=@

Find out raid array UUID:

sudo blkid -s UUID -o value /dev/md0
f2233ed8-fbe6-4199-890a-d416c7a2a62b

Last edited by tz4678 (2023-10-27 00:32:25)