Arch Linux

krupd

Member

Registered: 2018-11-30

Posts: 4

Drive decryption fails on kernel 4.20.6+

Hi there,

After an upgrade to kernel linux 4.20.6.arch1-1 I am unable to decrypt my root partition and finish booting. I am using the LVM on LUKS method of encryption with an unencrypted boot partition. The command used to setup the encryption was:

cryptsetup -v --type luks --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat device

The error I receive is:

device-mapper: reload ioctl on  failed: Invalid argument
Failed to setup dm-crypt key mapping for device /dev/disk/by-uuid/UUID
Check that kernel supports aes-xts-plain-64 cipher (check syslog for more info)
Failed to activate: input/output error

I get the same error using the udev and encrypt hooks as I do with the systemd based hooks. Downgrading to kernel 4.20.5 allows me to boot normally. I am running on an AMD Threadripper 1950x based system.

Last edited by krupd (2019-02-10 04:47:44)

23pieces

Member

Registered: 2019-02-08

Posts: 12

Re: Drive decryption fails on kernel 4.20.6+

Hey dude!

I got the same error here. Downgrading was also the only solution for me.
I also testet the hardened kernel >= 4.20.6 and the lts kernel, neither work.

Also i opened a task in the bugtracker:
https://bugs.archlinux.org/task/61657?p … &sort=desc

More interestingly it seems that it doesnt occur on every system.
I tried to setup a VM with similar settings and it's working.

I also found that they changed something in dm-crypt in the 4.20.6 changelog
https://cdn.kernel.org/pub/linux/kernel … Log-4.20.6    search for dm-crypt

Tho i'm not sure that this has to do something with the issue, just a guess.

See also my post:
https://bbs.archlinux.org/viewtopic.php?id=244057

Last edited by 23pieces (2019-02-08 10:04:04)

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: Drive decryption fails on kernel 4.20.6+

The following is the dm crypt change from 4.20.6 https://git.kernel.org/pub/scm/linux/ke … 54f03c6bf8
Seems unlikely to be the cause but if you revert it and it resolves the issue that would prove it is the cause.  If that fails please bisect between 4.20.5 and 4.20.6.

23pieces

Member

Registered: 2019-02-08

Posts: 12

Re: Drive decryption fails on kernel 4.20.6+

I can confirm the commit is the cause of the error, i also talked with the developers who made the commit.
I'm working on a solution ( if there is one besides reinstall) and also on a detailed explanation of the problem and why this problem occurs.

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: Drive decryption fails on kernel 4.20.6+

Is there a mailing list or bug tracker reference?

23pieces

Member

Registered: 2019-02-08

Posts: 12

Re: Drive decryption fails on kernel 4.20.6+

The thing is, as i understood, it is not a bug. They HAD a bug and they fixed it. It was a bug in the dm-crypt parser.
They just ignored the error. But now they dont ignore it anymore that's why this error occurs.

The problem is that you have all that stuff in the luksheader and now the parser says "nope thats wrong".

Please give me some time to prepare something more detailed (Maybe tomorrow or on sunday).

Concerning mailing list or bug tracker: I wrote with Milan Broz and Greg Kroah-Hartman directly (bad idea i know).

Last edited by 23pieces (2019-02-08 19:02:17)

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: Drive decryption fails on kernel 4.20.6+

No problem I was just looking through the dm-devl archive I could only find the original issue which as you noted this is a fix for https://www.redhat.com/archives/dm-deve … 00055.html
As the commit only landed in v5.0-rc4 it appears to be a regression to me for v5.0 and I would suggest the commit should be reverted from mainline and stable until it is resolved.

frostschutz

Member

Registered: 2013-11-15

Posts: 1,475

Re: Drive decryption fails on kernel 4.20.6+

I'd love to help but it works just fine for me. ArchLinux running 4.20.6-arch1-1-ARCH kernel, using LUKS with aes-xts-plain64 as described in first post, not an issue whatsoever.

If it stopped working for everyone I expect a lot more noise in the forums...? Lots of people use LUKS after all.

How to reproduce the bug?

----

Also the error message suggests it could be missing module or other reasons. With initramfs you always have to check (lsmod, cat /proc/crypto, lsinitcpio, ...compare differences).

Does it say anything in dmesg?

Last edited by frostschutz (2019-02-08 19:35:49)

23pieces

Member

Registered: 2019-02-08

Posts: 12

Re: Drive decryption fails on kernel 4.20.6+

krupd can you paste the output of:

cryptsetup luksDump /dev/disk/by-uuid/UUID

Please

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: Drive decryption fails on kernel 4.20.6+

23pieces can you not provide your reproducer?

23pieces

Member

Registered: 2019-02-08

Posts: 12

Re: Drive decryption fails on kernel 4.20.6+

Make a small image:
dd if=/dev/zero bs=4M count=20 > crypt.img

Setup a loop device for testing:
losetup /dev/loop2 crypt.img

Now we have a loop device where we can play with cryptsetup.

Try on kernel BEFORE 4.20.6:

cryptsetup -v --type luks --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat /dev/loop2

cryptsetup luksDump /dev/loop2
LUKS header information for /dev/loop2

Version:           1
Cipher name:       aes
Cipher mode:       xts-plain64
Hash spec:         sha512
Payload offset:    4096
MK bits:           512

2nd:

cryptsetup -v --type luks --cipher aes-xts-plain-64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat /dev/loop2

Be aware of the difference here "aes-xts-plain-64". Now we have the "-" between plain and 64.

cryptsetup luksDump /dev/loop2
LUKS header information for /dev/loop2

Version:           1
Cipher name:       aes
Cipher mode:       xts-plain-64
Hash spec:         sha512
Payload offset:    4096
MK bits:           512

We now have a cipher mode xts-plain-64 in the LUKS Header, which in reality doesnt exist.

Try on kernel at LEAST 4.20.6:

Try both cryptsetup commands from up.

In both cases u should get:

cryptsetup luksDump /dev/loop2
LUKS header information for /dev/loop2

Version:           1
Cipher name:       aes
Cipher mode:       xts-plain64
Hash spec:         sha512
Payload offset:    4096
MK bits:           512

What did they now do ( at least as i understood).

Before Kernel 4.20.6 they just stripped the -64 from aes-xts-plain-64 and used just used aes-xts-plain.
They suppressed the error in dm-crypt, but you should get a message in dmsg like:
device-mapper: crypt: Ignoring unexpected additional cipher options

The problem is: We now have a cipher in the luks header, which doesnt exist.

As i understood since the patch they dont strip anymore and they dont suppress the error anymore.

And thats why you can't boot.

Try to mount the luks partition you with xts-plain-64 under the 4.20.6 Kernel, doesnt work.

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: Drive decryption fails on kernel 4.20.6+

Thanks I can reproduce that.  Would that also not mean with plain64 being treated as plain with volume greater than 2TiB the IV will repeat which can potentially leak data?

frostschutz

Member

Registered: 2013-11-15

Posts: 1,475

Re: Drive decryption fails on kernel 4.20.6+

Oooh, I completely missed that. aes-xts-plain—64 instead of aes-xts-plain64.

# hexdump -C -n 64 crypt.img 
00000000  4c 55 4b 53 ba be 00 01  61 65 73 00 00 00 00 00  |LUKS....aes.....|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  78 74 73 2d 70 6c 61 69  |........xts-plai|
00000030  6e 2d 36 34 00 00 00 00  00 00 00 00 00 00 00 00  |n-64............|
00000040

Fix: (warning - for LUKS1 header only, not LUKS2! make LUKS Header Backup before trying this.)

printf "%s\0\0\0\0" xts-plain | dd conv=notrunc bs=1 seek=$((0x28)) of=crypt.img

Result:

# hexdump -C -n 64 crypt.img 
00000000  4c 55 4b 53 ba be 00 01  61 65 73 00 00 00 00 00  |LUKS....aes.....|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  78 74 73 2d 70 6c 61 69  |........xts-plai|
00000030  6e 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |n...............|
00000040

If the device is SMALLER THAN 2TB, then it's better to use plain64:

printf "%s\0\0\0\0" xts-plain64 | dd conv=notrunc bs=1 seek=$((0x28)) of=crypt.img

However if it's LARGER THAN 2TB then data after 2TB would no longer be decrypted correctly. (xts-plain and xts-plain64 are identical for the first 2TB only)

Alternative method to build new LUKS header from scratch using the same master key: https://unix.stackexchange.com/a/178722/30851 (this example was to change the hash used for the passphrase, but changing the cipher (without really changing it) is also possible this way.

Last edited by frostschutz (2019-02-08 22:57:11)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,475

Re: Drive decryption fails on kernel 4.20.6+

Would that also not mean with plain64 being treated as plain with volume greater than 2TiB the IV will repeat which can potentially leak data?

Yes, for volume >2TB you'd have to re-encrypt (anything that's past 2TB anyway) to  move plain to plain64.

It's possible to do with cryptsetup-reencrypt tool.

(If you don't want to muck with the LUKS header yourself, you can also use that to "fix" the broken header).

However, re-encryption a) takes a very long time b) is still very risky, any problem in the process and it's very hard to recover with one half of data encrypted this way and other half that way.

Last edited by frostschutz (2019-02-08 22:55:48)

23pieces

Member

Registered: 2019-02-08

Posts: 12

Re: Drive decryption fails on kernel 4.20.6+

Thx guys for the help!

I will try playing around with the headers today.
Should we write something for the wiki?

krupd

Member

Registered: 2018-11-30

Posts: 4

Re: Drive decryption fails on kernel 4.20.6+

krupd can you paste the output of:

cryptsetup luksDump /dev/disk/by-uuid/UUID

Please

Sorry, I was for some reason unsubscribed to this thread. My luksDump is as follows:

 LUKS header information for /dev/disk/by-uuid/4c4fad5f-5d8c-479e-abc1-38dfa62102ed

 Version:        1
 Cipher name:    aes
 Cipher mode:    xts-plain-64
 Hash spec:      sha512
 Payload offset: 4096
 MK bits:        512
 MK digest:      da 5a 1b 97 3b af 3b 2c 66 c7 12 66 b1 e8 64 34 84 b4 8a 27
 MK salt:        1a 83 51 4f ce 4f 69 09 8b 5b fd 92 1b 05 d8 3b
                 06 a9 7b f5 5e 7d 9d cd e3 8f 29 46 26 ca 85 90
 MK iterations:  138994
 UUID:           4c4fad5f-5d8c-479e-abc1-38dfa62102ed
 
 Key Slot 0: ENABLED
     Iterations:             2272104
     Salt:                   37 c4 8a 40 70 57 11 05 60 88 94 d9 d8 4e 0f 0e
                             dd 63 7c 39 f4 99 1a 56 e8 51 25 4a 18 07 ba 98
     Key material offset:    8
     AF stripes:             4000
 Key Slot 1: DISABLED
 Key Slot 2: DISABLED
 Key Slot 3: DISABLED
 Key Slot 4: DISABLED
 Key Slot 5: DISABLED
 Key Slot 6: DISABLED
 Key Slot 7: DISABLED

seems to be consistent with the problem you stated

frostschutz

Member

Registered: 2013-11-15

Posts: 1,475

Re: Drive decryption fails on kernel 4.20.6+

@krupd

Yeah, it's a typo in the Cipher mode, it's supposed to be xts-plain or xts-plain64, not xts-plain-64.

Depending on device size, change it to xts-plain (if larger than 2TB or unsure) or xts-plain64 (if smaller than 2TB) using the method I described above, in your case replace crypt.img with /dev/disk/by-uuid/4c4fad5f-5d8c-479e-abc1-38dfa62102ed