Arch Linux

dawnofman

Member

Registered: 2019-07-26

Posts: 140

[solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

I am running BTRFS (linux 5.17.5 with btrfs-progs 5.17-1 on Xeon CPUs) on a couple volumes which were previously formatted with --csum=crc32c; thus I have the following in my mkinitcpio.conf:

MODULES=(crc32c-intel)

So far so good.

Now I want to add more non-related drives/volumes to the existing ones and learned of the pros of better checksum algos like xxhash[64] and sha256. Since I am not planning to use dedup for the time being I think the best choice is xxhash[64] vs crc32c which is 32-bit:

- https://kdave.github.io/btrfs-hilights-5.5-new-hashes/
- our wiki entry for BTRFS has no mention of xxhash by the way

This means that I will end having some volumes with crc32c and some others with xxhash at the same time.

Would either of the following mkinitcpio entries be correct ?

MODULES=(crc32c-intel, xxhash)

or

MODULES=(crc32c-intel, xxhash64)

Can two hashes coexist ? Any trouble/issues/suggestions you are aware of ?

If they can I suppose I should rebuild the initramfs before start formatting with --csum=xxhash in order to have hardware acceleration already in place.

If not I will need to reformat my system/home volumes with xxhash instead of crc32c ending up in a one-man-show deployment with xxhash.

Last edited by dawnofman (2022-06-13 21:35:09)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 21,743

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

they should be able to coexist xxhash is already a builtin which might already be enough, other than that the correct line would be

MODULES=(crc32c-intel  xxhash64)

no comma just a space

dawnofman

Member

Registered: 2019-07-26

Posts: 140

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

they should be able to coexist xxhash is already a builtin which might already be enough, other than that the correct line would be

MODULES=(crc32c-intel  xxhash64)

no comma just a space

Thanks for your reply V1del. You're right, the comma was my typo. I'll build my initramfs images adding xxhash64 then. Thanks.

Raynman

Member

Registered: 2011-10-22

Posts: 1,539

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

I see only xxhash_generic in Arch kernel packages? And mkinitcpio should just pick these up as dependencies of the btrfs module.

loqs

Member

Registered: 2014-03-06

Posts: 17,378

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

I see only xxhash_generic in Arch kernel packages?

xxhash64 is an alias for xxhash_generic https://github.com/torvalds/linux/blob/ … ric.c#L105
btrfs uses the alias xxhash64 rather than xxhash_generic https://github.com/torvalds/linux/blob/ … er.c#L2747

Raynman

Member

Registered: 2011-10-22

Posts: 1,539

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

Right. btrfs naturally also refers to crc32c, not crc32c-intel. There shouldn't be a problem with putting that in MODULES, but the name doesn't suggest any specific hardware acceleration.

loqs

Member

Registered: 2014-03-06

Posts: 17,378

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

Right. btrfs naturally also refers to crc32c, not crc32c-intel. There shouldn't be a problem with putting that in MODULES, but the name doesn't suggest any specific hardware acceleration.

True the xxhash_generic generic module is a thin wrapper around the xxhash module that Arch ships as a builtin,  which has no hardware acceleration.

dawnofman

Member

Registered: 2019-07-26

Posts: 140

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

Right. btrfs naturally also refers to crc32c, not crc32c-intel. There shouldn't be a problem with putting that in MODULES, but the name doesn't suggest any specific hardware acceleration.

True the xxhash_generic generic module is a thin wrapper around the xxhash module that Arch ships as a builtin,  which has no hardware acceleration.

Correct me if I am wrong: the only hardware-accelerated hash for Intel Xeon E5-#### v3 CPUs is SHA256 meaning AES-NI acceleration ?

loqs

Member

Registered: 2014-03-06

Posts: 17,378

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

Correct me if I am wrong: the only hardware-accelerated hash for Intel Xeon E5-#### v3 CPUs is SHA256 meaning AES-NI acceleration ?

https://github.com/torvalds/linux/blob/ … lue.c#L418
SHA256 has implementations using SSE3,  AVX,  AVX2 and AES-NI.

https://github.com/torvalds/linux/blob/ … hash.c#L55
https://github.com/torvalds/linux/blob/ … glue.c#L71
blake2s-256 has implementations using SSE3 and AVX512

https://github.com/Cyan4973/xxHash/issues/62 covers xxhash vs crc32c
https://github.com/Cyan4973/xxHash/issues/546 also covers the hashes of interest

https://github.com/torvalds/linux/blob/ … _glue.c#L5
crc32c-intel uses SSE4.2

Last edited by loqs (2022-06-15 01:21:40)

dawnofman

Member

Registered: 2019-07-26

Posts: 140

Re: [solved] can crc32c and xxhash coexist in mkinitcpio (for btrfs) ?

Correct me if I am wrong: the only hardware-accelerated hash for Intel Xeon E5-#### v3 CPUs is SHA256 meaning AES-NI acceleration ?

https://github.com/torvalds/linux/blob/ … lue.c#L418
SHA256 has implementations using SSE3,  AVX,  AVX2 and AES-NI.

https://github.com/torvalds/linux/blob/ … hash.c#L55
https://github.com/torvalds/linux/blob/ … glue.c#L71
blake2s-256 has implementations using SSE3 and AVX512

https://github.com/Cyan4973/xxHash/issues/62 covers xxhash vs crc32c
https://github.com/Cyan4973/xxHash/issues/546 also covers the hashes of interest

https://github.com/torvalds/linux/blob/ … _glue.c#L5
crc32c-intel uses SSE4.2

Thanks for your very-specific links loqs -those helped me a lot !

So checking Intel Xeon E5-2630 v3 specifications:

- have AES-NI
- have AVX2

... those are my only hardware-accelerated options and both have implementations as stated on the link you provided: SHA-256

Of interest on one of the links you provided:

xxHash was created in a different context, using a cpu without this capability, and with intended goal of maximum portability, well beyond Intel's realm (arm, mips, power, etc.). Hence no reliance on brand-specific features.

And also of interest on the other one:

eg: for a i7 2600K processor:

     SHA-256:   103.15 MB/s (done     1.01 GB)
      BLAKE3:   152.51 MB/s (done     1.49 GB) +1.48x
       SHA-1:   348.43 MB/s (done     3.40 GB)
    XXHASH64:     2.93 GB/s (done    29.30 GB)
        XXH3:     3.11 GB/s (done    31.02 GB)
     CRC-32C:     3.19 GB/s (done    31.89 GB)
      WYHASH:     3.88 GB/s (done    38.75 GB)
      CRC-32:     4.07 GB/s (done    40.63 GB)

Setting aside the actual throughput stated for each of those processors (there are more ones on the link) the relevant fact here is that both SHA-256 and BLAKE-3 being "true-crypto" algos are far far faaar slower than CRC-32 or XXHASH-64; they are on a totally different league; which makes me wonder what could be the actual impact for real-life usage on a SSD.

The choice of XXHASH-64 over CRC-32C for BTRFS is a no-brainer.

It seems the choice of far-stronger algos just to keep track of BTRFS chunk hash-sums is a more-elaborate one.

Performance-wise we are talking an order of magnitude up/down depending the hashsum choice.

Last edited by dawnofman (2022-06-15 01:59:22)