You are not logged in.

#1 2022-01-08 13:35:00

ivanoff
Member
Registered: 2012-03-18
Posts: 66

Fstrim (periodic trim) and SSD raid (raid5)

Hello,
Fstrim doesn't apparently work with RAID SSDs (soft raid5 with mdadm). I've read several articles on the subject but none were very conclusive.
I wonder whether periodic trim can be applied to SSD RAID?
If not, is it so big an issue? Is it worth it anyway?
What do you think?

Thank you
Cheers

Offline

#2 2022-01-08 14:58:26

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Fstrim (periodic trim) and SSD raid (raid5)

It should work, mdadm raid has supported trim for a long time now (maybe not in all levels and layouts, but standard ones should work). (EDIT: I was wrong, as @progandy pointed out below, there is a safety flag for the raid5 case)

How are you testing it? Any error messages in particular?

If you have LUKS in between, that's usually what prevents trim unless you open with allow discard flag.

Last edited by frostschutz (2022-01-08 19:58:37)

Offline

#3 2022-01-08 15:34:49

ivanoff
Member
Registered: 2012-03-18
Posts: 66

Re: Fstrim (periodic trim) and SSD raid (raid5)

I am checking fstrim.timer output: journalctl -u fstrim

 
janv. 03 19:49:57 nordi systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
janv. 03 19:49:59 nordi fstrim[60198]: /boot : 50,2 GiB (53878419456 octets) réduits sur /dev/sdb1
janv. 03 19:49:59 nordi systemd[1]: fstrim.service: Deactivated successfully.
janv. 03 19:49:59 nordi systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.

Only my boot partition gets trimmed. My raid /dev/md0 is mounted on /
I dont use LUKS

Offline

#4 2022-01-08 15:44:02

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Fstrim (periodic trim) and SSD raid (raid5)

Maybe it's not in /etc/fstab?

Also check lsblk, lsblk --discard

Offline

#5 2022-01-08 16:44:19

ivanoff
Member
Registered: 2012-03-18
Posts: 66

Re: Fstrim (periodic trim) and SSD raid (raid5)

Yes, it's in fstab:
UUID=ddfcab13-4845-4728-ba5e-87860092abb3       /               ext4            defaults,relatime       0 1

I don't know how to interpret the result of lsblk though:

NAME    DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda            0      512B       2G         0
└─sda1         0      512B       2G         0
  └─md0        0       32K       2G         0
sdb            0      512B       2G         0
├─sdb1         0      512B       2G         0
└─sdb2         0      512B       2G         0
sdc            0      512B       2G         0
└─sdc1         0      512B       2G         0
  └─md0        0       32K       2G         0
sdd            0      512B       2G         0
└─sdd1         0      512B       2G         0
  └─md0        0       32K       2G         0

Does this mean, trim should work?
However,

fstrim -v /
fstrim: /: the discard operation is not supported

Offline

#6 2022-01-08 16:58:02

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Fstrim (periodic trim) and SSD raid (raid5)

You need to check if your ssds can handle discards in an mdadm  compatible way:
https://www.kernel.org/doc/html/latest/ … rd-support
Then add raid456.devices_handle_discards_safely=y to your kernel parameters.

Last edited by progandy (2022-01-08 17:24:32)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#7 2022-01-08 17:51:24

ivanoff
Member
Registered: 2012-03-18
Posts: 66

Re: Fstrim (periodic trim) and SSD raid (raid5)

Yes, that's it, you found it. It's certainly disabled by default.
Last question: I don't know where to look to find "Storage devices that properly support ‘discard_zeroes_data’ " ? Is there a SSD catalogue somewhere or a command to tell us?

Offline

#8 2022-01-08 18:01:11

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Fstrim (periodic trim) and SSD raid (raid5)

Probably like this: https://wiki.polaire.nl/doku.php?id=linux_test_ssd_trim And then you have to trust that i doesn't behave differently in e.g. a high stress situation.

Last edited by progandy (2022-01-08 18:01:54)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#9 2022-01-08 18:08:31

ivanoff
Member
Registered: 2012-03-18
Posts: 66

Re: Fstrim (periodic trim) and SSD raid (raid5)

OK, will do that, thank you. Works for a single device though. My raid is already assembled.

Last edited by ivanoff (2022-01-08 18:22:19)

Offline

#10 2022-01-08 19:18:51

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Fstrim (periodic trim) and SSD raid (raid5)

progandy wrote:

https://www.kernel.org/doc/html/latest/ … rd-support
Then add raid456.devices_handle_discards_safely=y to your kernel parameters.

Thanks for pointing it out, I was unaware of this change.

Even so this should be a documentation typo (discards -> discard) and would not have any effect unless specified correctly.

static bool devices_handle_discard_safely = false;
module_param(devices_handle_discard_safely, bool, 0644);
MODULE_PARM_DESC(devices_handle_discard_safely,
                 "Set to Y if all devices in each array reliably return zeroes on reads from discarded regions");

In order to test, after enabling this flag (disable fstrim service timer beforehand for this so it does not run accidentally), you should be using a filesystem that has already been populated with data and ideally been in use for a while.

Then run an mdadm check and verify mismatch_cnt is 0.

head /sys/block/md0/md/mismatch_cnt
mdadm --action=check /dev/md0
mdadm --wait /dev/md0
head /sys/block/md0/md/mismatch_cnt

If the result for mismatch_cnt is 0, you can try your luck with discard: (back up all your files first, just in case)

fstrim -v /

Then re-run the check above and verify the mismatch_cnt is still 0.

If in doubt you can also drop caches in between (echo 3 > /proc/sys/vm/drop_caches) as the Linux cache tends to return previously cached data that has already been discarded by the storage device.

If mismatches appear, then using discard on this raid array might not be entirely safe (in a subsequent disk failure and recovery case which is after all the point of raid).

In that case you can follow it up with a repair instead of a check to restore parity, and periodically discard your data in this fashion (backup, check, discard, repair).

Or simply make do without discard. People use fstrim/discard because everyone uses it, not because they actually need it...

Last edited by frostschutz (2022-01-08 19:38:54)

Offline

#11 2022-01-09 12:44:02

ivanoff
Member
Registered: 2012-03-18
Posts: 66

Re: Fstrim (periodic trim) and SSD raid (raid5)

Or, since I can probably live without automatic trim on the raid disks, I think I will wait for the function to be stable and safe enough to be enabled by default in the kernel...
Because the fstrim can work a few times with mismatch_cnt=0 and oneday generate parity errors I suppose... so hardware manufacturers: get your thing straight!

Offline

#12 2024-03-03 09:19:36

malvika.waes
Member
Registered: 2024-03-03
Posts: 3

Re: Fstrim (periodic trim) and SSD raid (raid5)

My drives report: "Write Zeroes Not Supported". Does this mean that it's not safe to use TRIM with a mdadm RAID6 array, or does it mean that it will just be less optimal? If the drive is explicitly saying it doesn't return zeros for trimmed blocks, then I would expect that mdadm would work correctly with that assumption. Is that correct?

Offline

#13 2024-03-03 10:10:43

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Fstrim (periodic trim) and SSD raid (raid5)

The current mode of operation seems to be "don't support discard at all" (for raid 4 / 5 / 6) and push responsibility to the user (require them to set a flag to re-enable discard support). I haven't followed mdadm development recently, so I'm not sure if there is any work to improve SSD support.

mdadm requires the array to be in sync at all times. If your drive returns random data after discard, it's not in sync.

To work around this, md would have to keep track of discarded regions in metadata, check parity with every write (at the cost of performance), get rid of optimizations that are based on the assumption that the array is always in sync (always recalculate parity from scratch instead of just updating it based on old data parity).

Comment in sourcecode drivers/md/raid5.c:

                /*
                 * zeroing is required, otherwise data
                 * could be lost. Consider a scenario: discard a stripe
                 * (the stripe could be inconsistent if
                 * discard_zeroes_data is 0); write one disk of the
                 * stripe (the stripe could be inconsistent again
                 * depending on which disks are used to calculate
                 * parity); the disk is broken; The stripe data of this
                 * disk is lost.
                 *
                 * We only allow DISCARD if the sysadmin has confirmed that
                 * only safe devices are in use by setting a module parameter.
                 * A better idea might be to turn DISCARD into WRITE_ZEROES
                 * requests, as that is required to be safe.
                 */

If in doubt, run a raid check, and verify mismatch_cnt = 0.

If there are mismatches, data will be corrupted on any rebuild - and rebuilds also happen after power loss, etc, not just on disk failure, so. In a way, it's worse than raid 0.

Last edited by frostschutz (2024-03-03 10:20:00)

Offline

#14 2024-03-04 11:19:27

malvika.waes
Member
Registered: 2024-03-03
Posts: 3

Re: Fstrim (periodic trim) and SSD raid (raid5)

ok, I anyway tried setting /sys/module/raid456/parameters/devices_handle_discard_safely = Y, (this is a testing system only), but when attempting fstrim, I was informed the operation is not supported. I guess mdadm is checking if /sys/block/md127/queue/discard_zeroes_data = 1, and not allowing it (since it's 0)

Last edited by malvika.waes (2024-03-04 11:19:56)

Offline

#15 2024-03-16 06:23:28

malvika.waes
Member
Registered: 2024-03-03
Posts: 3

Re: Fstrim (periodic trim) and SSD raid (raid5)

Just a thought - can't fstrim/mdadm be made to work on full stripes? So only if all the blocks for a full stripe can be deleted, then the discard is done on those blocks. Wouldn't this eliminate the need for any parity recalculation etc..? In our case we are using the RAID for a video editing server, which stores very large video files, which get replaced after a project is completed. I'm sure a large percentage of deletions would result in complete stripes being discarded, so even having this functionality would probably boost the write performance quite a lot, wouldn't it?

Offline

Board footer

Powered by FluxBB