You are not logged in.

#1 2019-09-09 18:55:38

killermoehre
Member
Registered: 2019-08-14
Posts: 4

device-mapper crypt AEAD ERROR

Hi,

after a hibernate my system is not coming up again, because the partition containing / can't be mounted on /sysroot in the initramfs. I use LVM on LUKS. The encryption works. / is on XFS. The error I get is

kernel: device-mapper: crypt: dm-1: INTEGRITY AEAD ERROR, sector 117442560
kernel: Buffer I/O error on dev dm-4, logical block 0, async page read
mount: /sysroot: mount(2) system call failed: Invalid or incomplete multibyte or wide character.

Any advice?

Offline

#2 2019-09-09 20:05:33

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: device-mapper crypt AEAD ERROR

Was the LUKS container created with integrity AEAD ?  If you reboot is the error still present?

Offline

#3 2019-09-10 06:08:43

killermoehre
Member
Registered: 2019-08-14
Posts: 4

Re: device-mapper crypt AEAD ERROR

loqs wrote:

Was the LUKS container created with integrity AEAD ?  If you reboot is the error still present?

Yes and yes.

Offline

#4 2019-09-10 09:19:32

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: device-mapper crypt AEAD ERROR

I would suggest from the install media running a S.M.A.R.T. self test and posting the results.

Offline

#5 2019-09-10 17:02:51

killermoehre
Member
Registered: 2019-08-14
Posts: 4

Re: device-mapper crypt AEAD ERROR

loqs wrote:

I would suggest from the install media running a S.M.A.R.T. self test and posting the results.

root@archiso ~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.20.0-arch1-1-ARCH] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZVPW256HEGL-00000
Serial Number:                      S346NY0J108747
Firmware Version:                   CXZ7300Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 256,060,514,304 [256 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256,060,514,304 [256 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 c1710052bc
Local Time is:                      Tue Sep 10 16:58:40 2019 UTC
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     73 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.80W       -        -    0  0  0  0        0       0
 1 +     5.60W       -        -    1  1  1  1        0       0
 2 +     4.40W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1500
 4 -   0.0050W       -        -    4  4  4  4     2200    6000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        23 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    5%
Data Units Read:                    3,937,489 [2.01 TB]
Data Units Written:                 10,654,473 [5.45 TB]
Host Read Commands:                 94,156,071
Host Write Commands:                77,406,554
Controller Busy Time:               1,926
Power Cycles:                       764
Power On Hours:                     535
Unsafe Shutdowns:                   115
Media and Data Integrity Errors:    0
Error Information Log Entries:      877
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               23 Celsius
Temperature Sensor 2:               30 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0        877     0  0x002c  0x4016  0x004            0     1     -
  1        876     0  0x0006  0x4004      -            0     0     -
  2        875     0  0x001c  0x4016  0x004            0     1     -
  3        874     0  0x0006  0x4004      -            0     0     -
  4        873     0  0x0008  0x4004      -            0     1     -
  5        872     0  0x0018  0x4004      -            0     1     -
  6        871     0  0x0008  0x4004      -            0     1     -
  7        870     0  0x0014  0x4004      -            0     -     -
  8        869     0  0x0014  0x4004      -            0     -     -
  9        868     0  0x0014  0x4004      -            0     -     -
 10        867     0  0x0014  0x4004      -            0     -     -
 11        866     0  0x0006  0x4004      -            0     0     -
 12        865     0  0x0006  0x4004      -            0     0     -
 13        864     0  0x0006  0x4004      -            0     0     -
 14        863     0  0x0006  0x4004      -            0     0     -
 15        862     0  0x0006  0x4004      -            0     0     -
... (48 entries not shown)

Offline

#6 2019-09-11 00:15:03

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: device-mapper crypt AEAD ERROR

I would suggest creating an image of the disk before continuing if you do not have a backup.
Then `xfs_repair -n /dev/device` to see what xfs_repair would do before removing -n to attempt repairing the filesystem.
The man page for xfs_repair does contain this warning

Disk Errors
       xfs_repair aborts on most disk I/O errors. Therefore, if you are trying
       to  repair  a  filesystem that was damaged due to a disk drive failure,
       steps should be taken to ensure that all blocks in the  filesystem  are
       readable and writable before attempting to use xfs_repair to repair the
       filesystem. A possible method is using dd(8) to copy the  data  onto  a
       good disk.

If dm-integrity returns a read error on checksum mismatch it may cause xfs_repair to fail as noted above.
In which case using badblocks or dd to rewrite those sectors may help.

Offline

#7 2019-09-11 16:17:15

killermoehre
Member
Registered: 2019-08-14
Posts: 4

Re: device-mapper crypt AEAD ERROR

loqs wrote:

I would suggest creating an image of the disk before continuing if you do not have a backup.
Then `xfs_repair -n /dev/device` to see what xfs_repair would do before removing -n to attempt repairing the filesystem.
The man page for xfs_repair does contain this warning

Disk Errors
       xfs_repair aborts on most disk I/O errors. Therefore, if you are trying
       to  repair  a  filesystem that was damaged due to a disk drive failure,
       steps should be taken to ensure that all blocks in the  filesystem  are
       readable and writable before attempting to use xfs_repair to repair the
       filesystem. A possible method is using dd(8) to copy the  data  onto  a
       good disk.

If dm-integrity returns a read error on checksum mismatch it may cause xfs_repair to fail as noted above.
In which case using badblocks or dd to rewrite those sectors may help.

xfs_repair fails with a missing superblock. I will try badblocks.

Offline

#8 2019-09-11 16:27:59

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: device-mapper crypt AEAD ERROR

If xfs_repair can not locate a valid secondary superblock I would suspect that filesystem is beyond repair but perhaps others more knowledgeable about XFS would know of ways to recover it.

Offline

#9 2019-09-11 18:14:39

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: device-mapper crypt AEAD ERROR

Do you actually have to recover this data or is it just an experiment?

The interesting question is, what data is in these sectors? Sure, it does not match the integrity checksum. But that doesn't necessarily mean the data is "bad" bad (as in useless for data recovery purposes).

Not sure how you're supposed to get the data anyway. Trying to reproduce such a situation artificially just gives read errors:

[root@ALU shm]# truncate -s 1G foobar.img

[root@ALU shm]# cryptsetup luksFormat foobar.img --integrity hmac-sha256

WARNING!
========
This will overwrite data on foobar.img irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for foobar.img: 
Verify passphrase: 
Wiping device to initialize integrity checksum.
You can interrupt this by pressing CTRL+c (rest of not wiped device will contain invalid checksum).
Finished, time 00:04.214,  941 MiB written, speed 223.3 MiB/s   
[root@ALU shm]# cryptsetup luksOpen foobar.img foobar
Enter passphrase for foobar.img: 
[root@ALU shm]# yes > /dev/mapper/foobar
yes: standard output: No space left on device
[root@ALU shm]# sync
[root@ALU shm]# dd bs=1M seek=500 skip=500 count=5 if=foobar.img of=barfoo.img
5+0 records in
5+0 records out
5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0082868 s, 633 MB/s
[root@ALU shm]# yes n > /dev/mapper/foobar
yes: standard output: No space left on device
[root@ALU shm]# sync
[root@ALU shm]# dd bs=1M seek=500 skip=500 count=5 conv=notrunc of=foobar.img if=barfoo.img
5+0 records in
5+0 records out
5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.00626197 s, 837 MB/s
[root@ALU shm]# echo 3 > /proc/sys/vm/drop_caches 
[root@ALU shm]# sync
[root@ALU shm]# hexdump -C /dev/mapper/foobar
00000000  6e 0a 6e 0a 6e 0a 6e 0a  6e 0a 6e 0a 6e 0a 6e 0a  |n.n.n.n.n.n.n.n.|
*
hexdump: /dev/mapper/foobar: Input/output error
1c000000

I created a 1G container, wrote yes to it, used dd to copy the yes-state for a 5M chunk, then wrote no to it, restored the 5M chunk [but not its integrity hashes, so it's invalid.

Bam you get a read error. But the data should be |y.y.y.y.y.y.y.|. How to get it though?

# dmsetup table | grep foobar
foobar_dif: 0 1927176 integrity 7:0 32768 32 J 5 journal_sectors:16368 interleave_sectors:32768 buffer_sectors:128 journal_watermark:50 commit_time:10000
foobar: 0 1927176 crypt capi:authenc(hmac(sha256),xts(aes))-plain64 :96:logon:cryptsetup:b6697942-f98e-4a7e-8f94-b33b131e5e38-d0 0 253:83 0 1 integrity:32:aead

Ah, what the heck is that. It looks very different from regular crypt mappings...

# cryptsetup luksDump --dump-master-key foobar.img 

WARNING!
========
Header dump with volume key is sensitive information
which allows access to encrypted partition without passphrase.
This dump should be always stored encrypted on safe place.

Are you sure? (Type uppercase yes): YES
Enter passphrase for foobar.img: 
LUKS header information for foobar.img
Cipher name:   	aes
Cipher mode:   	xts-plain64
Payload offset:	32768
UUID:          	b6697942-f98e-4a7e-8f94-b33b131e5e38
MK bits:       	768
MK dump:	15 18 d0 86 35 df db d9 4e 41 f2 b9 be c6 ed d4 
		05 7e 14 a2 c2 6e 75 5a 2c a4 30 ef 11 95 80 6f 
		1b dd eb bf 5f db ad 7b 72 47 46 15 72 0f 12 80 
		af c8 b8 aa 5c 28 11 60 27 48 65 17 b9 ea a3 50 
		87 ed 67 16 44 a7 6e 57 fb e5 87 5c 7c 3f 06 c0 
		3f 59 ff 73 30 23 18 1e 2a 4f 6f 78 79 db 2c 8e 

That should be the master key but it's weird. Normally it looks like this:

LUKS header information for barfoo.img
Cipher name:   	aes
Cipher mode:   	xts-plain64
Payload offset:	32768
UUID:          	e4ff7bae-34de-49ba-a836-98649caca26d
MK bits:       	512
MK dump:	c5 46 bf bd 9d 31 e9 af eb a9 80 38 87 6b b2 db 
		76 83 28 ab e3 20 fb b7 0b b1 53 43 3a fd 86 a3 
		6d fd 88 8a 4e de ff 67 20 c3 46 e7 87 50 f9 d3 
		39 39 3e 0a 82 3f bd 93 e9 39 bd dc 35 9f 98 8c 

So without integrity, the master key is a lot shorter

tl;dr if there's a way to ignore integrity, then I don't know it yet. Integrity adds 256 bits to the key. I'm not sure if it can simply be split off and ignored or if there's more to it.

Last edited by frostschutz (2019-09-11 18:16:43)

Offline

#10 2019-09-11 18:50:11

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: device-mapper crypt AEAD ERROR

frostschutz wrote:

tl;dr if there's a way to ignore integrity, then I don't know it yet. Integrity adds 256 bits to the key. I'm not sure if it can simply be split off and ignored or if there's more to it.

So after some experimenting I found that the aes master key is simply the first 512 bits, and I found one offset with data:

[root@ALU shm]# cryptsetup luksFormat foobar-test.img --master-key-file key_a --offset=51192         
WARNING: Device foobar-test.img already contains a 'crypto_LUKS' superblock signature.

WARNING!
========
This will overwrite data on foobar-test.img irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for foobar-test.img: 
Verify passphrase: 
[root@ALU shm]# cryptsetup luksOpen foobar-test.img  foobar-test
Enter passphrase for foobar-test.img: 
[root@ALU shm]# hexdump -C -n 16 /dev/mapper/foobar-test 
[root@ALU shm]# hexdump -C /dev/mapper/foobar-test  | head
00000000  6e 0a 6e 0a 6e 0a 6e 0a  6e 0a 6e 0a 6e 0a 6e 0a  |n.n.n.n.n.n.n.n.|
*
01000000  05 13 57 84 d1 27 a9 9f  a4 40 8e b2 59 41 f1 42  |..W..'...@..YA.B|
01000010  3a fc 2c 29 b4 0e c9 bf  b2 80 03 44 79 cf 1a 8c  |:.,).......Dy...|
01000020  45 1c e9 9f f7 e7 71 fd  48 88 b9 21 79 83 6d e1  |E.....q.H..!y.m.|

But it turns out integrity is some kind of interleaved format. Mapping it out block by block is possible, just not very practical...

no integrity address -> integrity address

0x01000000 -> 0x018ff000
0x02000000 -> 0x029ff000
0x03000000 -> 0x03aff000
...

Sorry for noise, as you can see, I have no practical experience with LUKS2+integrity yet. Maybe there's an easy way to get data even if integrity says it's the wrong data. Don't know.

Offline

#11 2019-09-11 19:04:52

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: device-mapper crypt AEAD ERROR

dm-integrity has a recovery mode. I don't know how to activate that if integrity was set up as part of LUKS, but the feature is marked as EXPERIMENTAL, so never use it without backups.


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

Board footer

Powered by FluxBB