Arch Linux

vecino

Member

Registered: 2022-12-10

Posts: 139

[SOLVED] Raid1 - btrfs or mdadm?

Hi, I would like to ask more experienced users / admins what you think about Raid1. I want to switch to btrfs filesystem, because for example the snapshots option with grub-btrfs is great.

I would like to know the opinion of others what you think is a better, more stable option and why?

1) Is it better to use "good old" mdadm with btrfs fs?

or

2) Use the native raid support directly in brtfs?

I tried both options in my virtual setup and both have some advantages and disadvantages. I don't know what to choose.

Please give me your opinion. Pros and cons.

Thanks

Last edited by vecino (2023-05-23 15:32:45)

schard

Forum Moderator

From: Hannover

Registered: 2016-05-06

Posts: 2,654

Website

Re: [SOLVED] Raid1 - btrfs or mdadm?

I have zero experience with btrfs.
In the company I work for, we had an office server where I used RAID1 on two SSDs with mdadm.
The FS I used was ext4.
It worked fine for us (until we migrated to an online nextcloud solution due to everyone working from home during Covid).
Just always remember, RAID (1) is not a backup. It just reduces the risk of total system failure in case one of the drives fail.

Last edited by schard (2023-04-22 11:03:47)

---

Inofficial first vice president of the Rust Evangelism Strike Force

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

I use btrfs raid1 just because I find it easier to deal with one level of abstraction instead of two, and because it's possible to convert every filesystem to raid online.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

@schard - "RAID (1) is not a backup" ... Yes, I am fully aware of that. I'm dealing with a router where data is almost worthless... and all I care about is that if one of the two drives fails, the router keeps routing.

@lunaryorn - For example, have you noticed that btrfs is more susceptible to power outages? Yes I have the system backed up on a UPS battery, but it can still happen. On the other hand, it's actually a feature of btrfs that it will eventually repair the data from the other drive. Have you ever solved such problems?

Furthermore, you need to remember to add "degraded" to the boot loader in the btrfs raid, so that the system boots atomically even with one disk if the other fails.

GRUB_CMDLINE_LINUX="rootflags=subvol=/@,degraded"

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

I don't think it's a good idea to add "degraded" to rootflags unconditionally.  This flag forces "mount" to ignore all raid-level constraints, which can silently hide configuration errors as well as real file system corruption.  If your disk array is degraded to a state where it refuses to mount without degraded I think you should absolutely not blindly mount it and boot from it. 

But for a router I'd probably not use RAID anyways.  Instead I'd just use in two cheap SD cards, one as readonly rootfs device and the other one for select persistent data, and with a bit of clever mounting and some glue you can have a system which only writes to disk very infrequently and only small amounts of data.  Even cheap SD cards survive something like 3000 full write cycles and realistically don't wear out from reading, so even if you mount your rootfs writable once a week for updates you're unlikely to see a disk failure for years.  And if keep data in tmpfs mostly, only write to the persistent data device selectively, aggressively unmount the data disk if unused, and use a flash-storage aware file system, wear leveling will probably keep that disk alive longer than the router itself, and even if it fails you'll still have a bootable system.   With pluggable storage you can also implement simple A/B update schemes by buying an additional SD card for updates.

This setup is likely an order of magnitude cheaper than a full raid setup, and still more reliable. For instance a power outage will by design never corrupt a read-only file system, so in worst case you'll have a corrupted data filesystem which comes down to a "factory reset" but keeps your router functional.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

I got my information from Wiki: https://wiki.archlinux.org/title/btrfs

If your root resides on the array, you must also add rootflags=degraded to your kernel parameters.

Without this parameter, if one of the two disks fails, the system will not boot and manual intervention is required:

mount /dev/sdaX /new_root -o subvol=@,degraded

And only then the system will boot with only one disk.

Or it can be handled through HOOK: https://github.com/yafengabc/btrfs-autodegraded (It's 9 years old but it works - tried and tested).

The router is built on Supermicro ITX HW with SSD drives. I will not put SD cards in it.

On these routers we have been using it for many years to our satisfaction only we have it on mdadm and the disks have Ext4 filesystem. Now I would like to convert it to BTRFS Raid.

So what do you think is the best practice in 2023? I found a lot of articles, but they are quite old.

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,652

Re: [SOLVED] Raid1 - btrfs or mdadm?

Sorry, I thought you were building your own home router from scratch.

If it's for some kind of appliance then I don't really understand what you're asking for. If dmraid is a successful setup satisfying your demands then why change? Just because you can? And if you can't

If you were wondering where your post went, you hit 'report', not 'reply'.  It happens

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

Eek, damn.  I'm sorry for the noise.

I got my information from Wiki: https://wiki.archlinux.org/title/btrfs

If your root resides on the array, you must also add rootflags=degraded to your kernel parameters.

Without this parameter, if one of the two disks fails, the system will not boot and manual intervention is required:

mount /dev/sdaX /new_root -o subvol=@,degraded

And only then the system will boot with only one disk.

These quotes come from a section about a specific issue from 2014 which advises degraded as a workaround to make the system boot at all. I would not understand these quotes as a general recommendation to add "degraded". Personally I think that's a dangerous thing to do.

The router is built on Supermicro ITX HW with SSD drives. I will not put SD cards in it.

As quoted, I'm sorry, I answered under the impression that you were looking to build your own home router from scratch. 

If you already have an appliance with a specific successful setup based on dmraid I'm not sure what you're looking for here with this broad open question.  I'd assume you have a certain set of requirements, and then your current setup either fulfils these so there'd not be a reason to change, or it doesn't, but then you'd have specific criteria to evaluate btrfs raid against.

Perhaps it'd help if you could outline why specifically you'd like to migrate to btrfs, so as to give specific criteria you're looking for.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

@lunaryorn As I wrote in my first post. I want to use Btrfs Snapshots and why use unnecessarily mdadm when Btrfs can raid natively. And I like to try new things. Trying new things improves us.

Personally I think that's a dangerous thing to do.

Well yeah, but as I described above, without the "degraded" parameter, if one disk fails, the system with one disk won't boot without my intervention. With degraded it does - I'm trying to simulate this in a virtual machine.

I want an ordinary router with snapshots, if there is a problem after an update, so that I can boot via grub-btrfs https://github.com/Antynea/grub-btrfs and make a fix from a snapshot.

I'm only writing here because I'm toying with the idea of scrapping mdadm radid and using native directly in btrfs nothing more.

I just want to know the opinions and experiences of more experienced admins than me.

Here's the advice to use degraded on the Proxmox forum: https://forum.proxmox.com/threads/syste … sk.108223/ it's Debian, but it's from 2022.

Last edited by vecino (2023-04-23 17:53:08)

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

@lunaryorn As I wrote in my first post. I want to use Btrfs Snapshots and why use unnecessarily mdadm when Btrfs can raid natively. And I like to try new things. Trying new things improves us.

Then just try it   Just do make sure you understand the differences, e.g. how btrfs raid1 is different from mdadm raid1, and whether that has any impact on your requirements.

Personally I think that's a dangerous thing to do.

Well yeah, but as I described above, without the "degraded" parameter, if one disk fails, the system with one disk won't boot without my intervention. With degraded it does - I'm trying to simulate this in a virtual machine.

It'll then also silently mount read-write if both disks fail partially, e.g. if both SSDs begin to wear out near their end of life.   If you write to a btrfs filesystem where all underlying devices are unhealthy you'd definitely risk data loss I believe.

I want an ordinary router with snapshots, if there is a problem after an update, so that I can boot via grub-btrfs https://github.com/Antynea/grub-btrfs and make a fix from a snapshot.

Mmh, this sounds more like a home router again.  In an appliance I'd have expected A/B updates with automatic rollback (e.g. using failed boot counting as in systemd-boot).

Here's the advice to use degraded on the Proxmox forum: https://forum.proxmox.com/threads/syste … sk.108223/ it's Debian, but it's from 2022.

Which ends up saying that you should not do this by default.

Last edited by 3beb6e7c46a615a (2023-04-23 18:23:04)

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

Mmh, this sounds more like a home router again.  In an appliance I'd have expected A/B updates with automatic rollback (e.g. using failed boot counting as in systemd-boot).

Please, any tip on wiki / or howto where I can read how to do it? I would like to learn something new. No this is not a home router, but yes it's probably not a pro solution, but it does what it's supposed to do and that's the most important thing.

I'll try to test it - I'm trying to simulate different situations and so far it's behaving well.

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

See https://btrfs.readthedocs.io/ and specifically https://btrfs.readthedocs.io/en/latest/ … hunks-raid for how btrfs profiles work.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

Ok thank you

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

@lunaryorn
Please are you using HOOKS or MODULES btrfs in /etc/mkinitcpio.conf ? Or systemd instead of udev ? I don't quite know what is the right solution.

From https://wiki.archlinux.org/title/btrfs

A workaround is to remove btrfs from the HOOKS array in /etc/mkinitcpio.conf and instead add btrfs to the MODULES array. Then regenerate the initramfs and reboot.

Another possible workaround is to remove the udev hook in mkinitcpio.conf and replace it with the systemd hook. In this case, btrfs should not be in the HOOKS or MODULES arrays.

Last edited by vecino (2023-04-25 18:10:41)

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

I use a systemd-based initrd.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

I don't like to bother, but could you please take a look at my mkinitcpio.conf file ? Is it ok like this or should it look different ?

# MODULES
MODULES=(btrfs)
 
# BINARIES
BINARIES=()
 
# FILES
FILES=()
 
# HOOKS
HOOKS=(base udev autodetect modconf kms keyboard keymap block filesystems fsck btrfs-autodegraded grub-btrfs-overlayfs)

HOOK btrfs-autodegraded = https://github.com/yafengabc/btrfs-autodegraded

I read https://wiki.archlinux.org/title/btrfs but it is not clear to me what is the best option in 2023.

The solution is to remove btrfs from the HOOKS field in the /etc/mkinitcpio.conf file and add btrfs to the MODULES field instead. Then regenerate initramfs and restart the computer.
Another possible solution is to remove the udev hook in the mkinitcpio.conf file and replace it with the systemd hook. In this case, btrfs should not be in the HOOKS or MODULES fields.

Thanks

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: [SOLVED] Raid1 - btrfs or mdadm?

As said above, I use a systemd-based initramfs; I do not know how to setup btrfs raid with a classic initrd.  And also as said, I'd never use "degraded" automatically.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

Ok please imagine a situation where for example you schedule a reboot after a kernel update for example at 3:00 in the morning ... at that moment one of the two disks fails and if I didn't use the HOOK with btrfs-autodegraded, the machine just wouldn't boot and would wait for me to physically access it / or connect via IPMI and make a repair.

This is actually the main reason in my case why I use raid1 at all. I don't care about the data on those drives they are worthless to me ... I just care about ensuring the reliability of the router, if one of the drives fails it keeps going without my intervention. And later I'll replace the faulty disk with a new one to make the array complete.

Anyway, thank you, we all have our own point of view. Systemd-based initramfs is still a big unknown for me as a new user and I'm using classic initrd for now.

vecino

Member

Registered: 2022-12-10

Posts: 139

Re: [SOLVED] Raid1 - btrfs or mdadm?

Please allow me one last question ... if I also want to use systemd within /etc/mkinitcpio.conf is this correct? Or should I do something different? Thank you

MODULES=()

BINARIES=()

FILES=()

HOOKS=(base systemd autodetect modconf block filesystems fsck)

https://wiki.archlinux.org/title/mkinitcpio