You are not logged in.

Pages: 1

#1 2019-05-21 00:57:33

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

[SOLVED] Corrupted LUKS volume after upgrade + reboot

TL;DR - https://bugs.archlinux.org/task/62693

I've been running a LUKS on LVM setup on my laptop successfully for over a year. After a upgrading & rebooting earlier today I can no longer access my encrypted home volume.

Relevant pacman.log:

[2019-05-20 13:56] [PACMAN] Running 'pacman --verbose --sync --refresh --sysupgrade'
[2019-05-20 13:56] [PACMAN] synchronizing package lists
[2019-05-20 13:56] [PACMAN] starting full system upgrade
[2019-05-20 14:02] [ALPM] transaction started
[2019-05-20 14:02] [ALPM] upgraded eog (3.32.1-1 -> 3.32.2-1)
[2019-05-20 14:02] [ALPM] upgraded file (5.36-1 -> 5.37-1)
[2019-05-20 14:02] [ALPM] upgraded gedit-plugins (3.32.0-1 -> 3.32.2-1)
[2019-05-20 14:02] [ALPM] upgraded liburcu (0.10.2-1 -> 0.11.0-1)
[2019-05-20 14:02] [ALPM] upgraded glusterfs (1:6.0-1 -> 1:6.1-1)
[2019-05-20 14:02] [ALPM-SCRIPTLET] -- Read https://gluster.readthedocs.io/en/latest/Upgrade-Guide/ page!
[2019-05-20 14:02] [ALPM] upgraded hugo (0.55.5-1 -> 0.55.6-1)
[2019-05-20 14:02] [ALPM] upgraded imagemagick (7.0.8.45-1 -> 7.0.8.46-1)
[2019-05-20 14:02] [ALPM] warning: /etc/libvirt/libvirtd.conf installed as /etc/libvirt/libvirtd.conf.pacnew
[2019-05-20 14:02] [ALPM] warning: /etc/libvirt/nwfilter/clean-traffic-gateway.xml installed as /etc/libvirt/nwfilter/clean-traffic-gateway.xm
l.pacnew
[2019-05-20 14:02] [ALPM] warning: /etc/libvirt/qemu.conf installed as /etc/libvirt/qemu.conf.pacnew
[2019-05-20 14:02] [ALPM] warning: /etc/libvirt/qemu/networks/default.xml installed as /etc/libvirt/qemu/networks/default.xml.pacnew
[2019-05-20 14:02] [ALPM] upgraded libvirt (5.2.0-1 -> 5.3.0-1)
[2019-05-20 14:03] [ALPM] upgraded linux-firmware (20190424.4b6cf2b-1 -> 20190514.711d329-1)
[2019-05-20 14:03] [ALPM] upgraded linux (5.1.2.arch1-1 -> 5.1.3.arch1-1)
[2019-05-20 14:03] [ALPM] upgraded linux-docs (5.1.2.arch1-1 -> 5.1.3.arch1-1)
[2019-05-20 14:03] [ALPM] upgraded linux-lts (4.19.43-1 -> 4.19.44-1)
[2019-05-20 14:03] [ALPM] upgraded polari (3.32.1-1 -> 3.32.2-1)
[2019-05-20 14:03] [ALPM] upgraded python-markdown (3.0.1-2 -> 3.1-1)
[2019-05-20 14:03] [ALPM] upgraded virtualbox-host-modules-arch (6.0.8-2 -> 6.0.8-3)
[2019-05-20 14:03] [ALPM] upgraded webkit2gtk (2.24.1-1 -> 2.24.2-1)
[2019-05-20 14:03] [ALPM] transaction completed
[2019-05-20 14:03] [ALPM] running '60-linux-lts.hook'...
[2019-05-20 14:03] [ALPM] running '60-linux.hook'...
[2019-05-20 14:03] [ALPM] running '90-linux-lts.hook'...
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Starting build: 4.19.44-1-lts
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [systemd]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-vconsole]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-lvm2]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-encrypt]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Image generation successful
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Starting build: 4.19.44-1-lts
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [systemd]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-vconsole]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-lvm2]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-encrypt]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts-fallback.img
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Image generation successful
[2019-05-20 14:03] [ALPM] running '90-linux.hook'...
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2019-05-20 14:03] [ALPM-SCRIPTLET] ==> Starting build: 5.1.3-arch1-1-ARCH
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [systemd]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-vconsole]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-lvm2]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [sd-encrypt]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2019-05-20 14:03] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Image generation successful
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Starting build: 5.1.3-arch1-1-ARCH
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [systemd]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [sd-vconsole]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [sd-lvm2]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [sd-encrypt]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2019-05-20 14:04] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> Image generation successful
[2019-05-20 14:04] [ALPM] running 'detect-old-perl-modules.hook'...
[2019-05-20 14:04] [ALPM] running 'glib-compile-schemas.hook'...
[2019-05-20 14:04] [ALPM] running 'gtk-update-icon-cache.hook'...
[2019-05-20 14:04] [ALPM] running 'pacorphans.hook'...
[2019-05-20 14:04] [ALPM-SCRIPTLET] ==> None found
[2019-05-20 14:04] [ALPM] running 'systemd-daemon-reload.hook'...
[2019-05-20 14:04] [ALPM] running 'systemd-sysctl.hook'...
[2019-05-20 14:04] [ALPM] running 'systemd-sysusers.hook'...
[2019-05-20 14:04] [ALPM] running 'systemd-tmpfiles.hook'...
[2019-05-20 14:04] [ALPM] running 'systemd-update.hook'...
[2019-05-20 14:04] [ALPM] running 'update-desktop-database.hook'...

I initially tried the LTS kernel and rolling back to 5.1.2 but saw the same issue with both kernels so dug a little deeper. (Note that I booted into an Ubuntu Live CD for GUI browser access so don't be thrown off by the older kernel versions. The results are the same when booting from an Arch Live USB.)

Relevant journal output:

...
May 20 21:56:08 thinkpad systemd[1]: systemd-rfkill.service: Succeeded.
May 20 21:57:32 thinkpad systemd[1]: dev-disk-by\x2duuid-04ed9a68\x2de7ab\x2d4b8d\x2d82a8\x2d449ea5dbc4bd.device: Job dev-disk-by\x2duuid-04ed9a68\x2de7ab\x2d4b8d\x2d82a8\x2d449ea5dbc4bd.device/start timed out.
May 20 21:57:32 thinkpad systemd[1]: Timed out waiting for device /dev/disk/by-uuid/04ed9a68-e7ab-4b8d-82a8-449ea5dbc4bd.
May 20 21:57:32 thinkpad systemd[1]: Dependency failed for Cryptography Setup for home.
May 20 21:57:32 thinkpad systemd[1]: Dependency failed for Local Encrypted Volumes.
May 20 21:57:32 thinkpad systemd[1]: cryptsetup.target: Job cryptsetup.target/start failed with result 'dependency'.
May 20 21:57:32 thinkpad systemd[1]: Dependency failed for /dev/mapper/home.
May 20 21:57:32 thinkpad systemd[1]: Dependency failed for /home.
May 20 21:57:32 thinkpad systemd[1]: Dependency failed for Local File Systems.
May 20 21:57:32 thinkpad systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
May 20 21:57:32 thinkpad systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
May 20 21:57:32 thinkpad systemd[1]: home.mount: Job home.mount/start failed with result 'dependency'.
May 20 21:57:32 thinkpad systemd[1]: Dependency failed for File System Check on /dev/mapper/home.
May 20 21:57:32 thinkpad systemd[1]: systemd-fsck@dev-mapper-home.service: Job systemd-fsck@dev-mapper-home.service/start failed with result 'dependency'.
May 20 21:57:32 thinkpad systemd[1]: dev-mapper-home.device: Job dev-mapper-home.device/start failed with result 'dependency'.
May 20 21:57:32 thinkpad systemd[1]: systemd-cryptsetup@home.service: Job systemd-cryptsetup@home.service/start failed with result 'dependency'.
May 20 21:57:32 thinkpad systemd[1]: dev-disk-by\x2duuid-04ed9a68\x2de7ab\x2d4b8d\x2d82a8\x2d449ea5dbc4bd.device: Job dev-disk-by\x2duuid-04ed9a68\x2de7ab\x2d4b8d\x2d82a8\x2d449ea5dbc4bd.device/start failed with result 'timeout'.
May 20 21:57:32 thinkpad systemd[1]: systemd-ask-password-wall.path: Succeeded.
...

Sure enough, there is no disk with that UUID when I list the contents of /dev/disk/by-uuid/. lsblk /dev/sda -o +UUID shows the following:

NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT UUID
sda              8:0    0 238.5G  0 disk             
├─sda1           8:1    0   550M  0 part             7C61-6D77
└─sda2           8:2    0   238G  0 part             yBqoOj-tLQU-KPV6-0PIA-KuP7-fbLJ-VtuS03
  ├─vg0-lvroot 253:0    0    24G  0 lvm              c8590a0e-dc8d-4f13-8563-197f12436739
  │ └─root     253:3    0    24G  0 crypt /mnt/root  e8d7a621-9508-4737-92ce-f2f27fead164
  └─vg0-lvhome 253:1    0   214G  0 lvm

A quick test shows a failure to read the LUKS header: cryptsetup --debug isLuks /dev/vg0/lvhome

# cryptsetup 2.0.2 processing "cryptsetup --debug isLuks /dev/vg0/lvhome"
# Running command isLuks.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/vg0/lvhome.
# Trying to open and read device /dev/vg0/lvhome with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/vg0/lvhome.
# Crypto backend (gcrypt 1.8.1) initialized in cryptsetup library version 2.0.2.
# Detected kernel Linux 4.15.0-29-generic x86_64.
# Loading LUKS2 header.
# Opening lock resource file /run/cryptsetup/L_253:1
# Acquiring read lock for device /dev/vg0/lvhome.
# Verifying read lock handle for device /dev/vg0/lvhome.
# Device /dev/vg0/lvhome READ lock taken.
# Trying to read primary LUKS2 header at offset 0.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 8192.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 16384.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 32768.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 65536.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 131072.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 262144.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 524288.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 1048576.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 2097152.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# Trying to read secondary LUKS2 header at offset 4194304.
# Opening locked device /dev/vg0/lvhome
# Veryfing locked device handle (bdev)
# LUKS2 header read failed (-22).
# Device /dev/vg0/lvhome READ lock released.
# Releasing crypt device /dev/vg0/lvhome context.
# Releasing device-mapper backend.
Command failed with code -1 (wrong or missing parameters).

The same test on the root volume is successful: cryptsetup --debug isLuks /dev/vg0/lvroot 

# cryptsetup 2.0.2 processing "cryptsetup --debug isLuks /dev/vg0/lvroot"
# Running command isLuks.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/vg0/lvroot.
# Trying to open and read device /dev/vg0/lvroot with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/vg0/lvroot.
# Crypto backend (gcrypt 1.8.1) initialized in cryptsetup library version 2.0.2.
# Detected kernel Linux 4.15.0-29-generic x86_64.
# PBKDF pbkdf2, hash sha256, time_ms 2000 (iterations 0), max_memory_kb 0, parallel_threads 0.
# Reading LUKS header of size 1024 from device /dev/vg0/lvroot
# Key length 64, device size 50331648 sectors, header size 4036 sectors.
# Releasing crypt device /dev/vg0/lvroot context.
# Releasing device-mapper backend.
Command successful.

cryptsetup has a repair option but it seems limited. I'm hoping for a better solution than restoring the entire home partition's filesystem from backup but am unsure what to try next. I'm looking for a backup LUKS header, but am unsure if I ever created one. Any suggestions are welcome.

Last edited by chr0mag (2019-05-21 23:21:55)

Offline

#2 2019-05-21 06:55:13

frostschutz
Member
Registered: 2013-11-15
Posts: 1,471

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

file -s? hexdump -C?

if the luks header is damaged that might have happened at any point while the machine was running... anything come to mind? unlikely for pacman to do it

Last edited by frostschutz (2019-05-21 06:56:30)

Offline

#3 2019-05-21 13:31:29

Bevan
Member
Registered: 2009-09-08
Posts: 100

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

Is that volume on an SSD and did you use the "discard" flag in your crypttab? We are currently hunting an issue with trim/discard on encrypted volumes in Linux 5.1 which may have caused the destruction: https://www.redhat.com/archives/dm-deve … 00084.html

Offline

#4 2019-05-21 14:00:18

frostschutz
Member
Registered: 2013-11-15
Posts: 1,471

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

ouch :-(

time to chmod -x fstrim

Offline

#5 2019-05-21 16:53:45

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

frostschutz wrote:

file -s? hexdump -C?

file -s

file -s /dev/vg0/lvhome
/dev/vg0/lvhome: symbolic link to ../dm-1
file -s /dev/dm-1
/dev/dm-1: data
file -s /dev/vg0/lvroot
/dev/vg0/lvroot: symbolic link to ../dm-0
file -s /dev/dm-0
/dev/dm-0: LUKS encrypted file, ver 1 [aes, xts-plain64, sha256] UUID: c8590a0e-dc8d-4f13-8563-197f12436739

hexdump -C

dd if=/dev/vg0/lvroot count=1 | hexdump -C
00000000  4c 55 4b 53 ba be 00 01  61 65 73 00 00 00 00 00  |LUKS....aes.....|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  78 74 73 2d 70 6c 61 69  |........xts-plai|
00000030  6e 36 34 00 00 00 00 00  00 00 00 00 00 00 00 00  |n64.............|
00000040  00 00 00 00 00 00 00 00  73 68 61 32 35 36 00 00  |........sha256..|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 10 00 00 00 00 40  |...............@|
00000070  29 66 45 f4 84 41 3a 80  4f 2e 76 e9 d8 e4 d5 d1  |)fE..A:.O.v.....|
00000080  ea 84 d2 75 06 e7 9c 4f  37 fe e3 e7 bd 0a 23 70  |...u...O7.....#p|
00000090  ef 26 9a 3e 05 5a 9e 7a  4d a5 12 bf c9 86 40 bc  |.&.>.Z.zM.....@.|
000000a0  e9 42 42 68 00 01 4c e6  63 38 35 39 30 61 30 65  |.BBh..L.c8590a0e|
000000b0  2d 64 63 38 64 2d 34 66  31 33 2d 38 35 36 33 2d  |-dc8d-4f13-8563-|
000000c0  31 39 37 66 31 32 34 33  36 37 33 39 00 00 00 00  |197f12436739....|
000000d0  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 08 00 00 0f a0  |................|
00000100  00 ac 71 f3 00 14 a5 28  1e 45 ea 51 9f 47 63 94  |..q....(.E.Q.Gc.|
00000110  7e ea bf 8e 84 d4 61 a7  53 7f 06 bb 28 e0 0e a0  |~.....a.S...(...|
00000120  31 0d 34 9f 19 3f 52 78  00 00 02 00 00 00 0f a0  |1.4..?Rx........|
00000130  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 03 f8 00 00 0f a0  |................|
00000160  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 05 f0 00 00 0f a0  |................|
00000190  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 07 e8 00 00 0f a0  |................|
000001c0  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 09 e0 00 00 0f a0  |................|
000001f0  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200
dd if=/dev/vg0/lvhome count=1 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

The LUKS header for my home volume looks well and truly gone. In fact you have to get well into the file before seeing any data at all.

dd if=/dev/vg0/lvhome count=4089 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
001ff000  ee ee 6e ea 0c f9 81 0c  bb 5a cc 7c 2b c0 e8 78  |..n......Z.|+..x|
001ff010  8b f1 01 e9 3e 85 0f 9e  51 5f 9e d3 08 a4 be 8f  |....>...Q_......|
001ff020  fa 1d 92 86 2a d5 a3 48  42 90 6e a2 b6 90 92 77  |....*..HB.n....w|
001ff030  56 d4 e7 65 4b b8 d4 42  e2 92 51 b5 21 67 86 61  |V..eK..B..Q.!g.a|
001ff040  19 e3 bf ce df ac 8e 5f  8d 51 f8 0a af eb 4b 4e  |......._.Q....KN|
001ff050  3c a7 1c 92 8c c1 2f b5  01 42 92 65 ed 37 00 e9  |<...../..B.e.7..|
001ff060  da 2d 5d ef 87 63 aa 40  2b 6b 31 1f 31 4d f2 cd  |.-]..c.@+k1.1M..|
001ff070  01 b8 15 39 00 06 c3 ee  8d c6 e0 20 80 15 be 00  |...9....... ....|
001ff080  25 aa b3 fb af 23 69 26  25 46 7c 9d 13 fd e3 f3  |%....#i&%F|.....|
001ff090  69 fc d6 ae 37 75 a3 86  b0 fa a5 42 2e 59 ae 19  |i...7u.....B.Y..|
001ff0a0  a0 02 b0 89 af d0 1b a0  70 f5 d8 bb e5 4b aa 4c  |........p....K.L|
001ff0b0  3c 2a 61 dc d5 97 ef 79  81 e6 fc 85 6b e3 40 3b  |<*a....y....k.@;|
001ff0c0  ce 46 96 ce 4e d6 95 6d  97 00 b4 5a ef 7e 1a 95  |.F..N..m...Z.~..|
001ff0d0  29 8c 21 c9 75 a8 e9 4e  aa ec 79 bf d4 db 1e 0f  |).!.u..N..y.....|
001ff0e0  34 19 6e 69 a9 39 fa d1  20 9f f6 60 17 b3 16 24  |4.ni.9.. ..`...$|
001ff0f0  19 d4 f8 46 91 5c 3f ed  0a 1e 96 5a 36 93 82 1d  |...F.\?....Z6...|
001ff100  a0 2c 9a 46 94 c3 11 d8  ab 89 a3 06 74 17 87 46  |.,.F........t..F|
001ff110  02 05 87 73 18 be d5 8d  1c 2e 94 06 68 a3 8e d8  |...s........h...|
001ff120  0b 1d 36 1b bd 12 75 5d  62 1f db 26 43 db 63 a3  |..6...u]b..&C.c.|
001ff130  cc 94 b5 97 a9 12 67 51  81 68 7e fc e7 a9 60 7e  |......gQ.h~...`~|
001ff140  64 ae 3e 17 c7 92 53 48  92 bb 22 d6 99 81 8f 06  |d.>...SH..".....|
001ff150  be 47 6f d2 e1 53 cd 5c  9d 12 22 96 ee 08 8c f4  |.Go..S.\..".....|
001ff160  1c 75 36 b6 fd e5 4c 3e  6a 20 70 96 ed 88 33 e9  |.u6...L>j p...3.|
001ff170  6f c8 45 05 1f 8d 8c e1  5a 1a df a3 bd 7e 7d d8  |o.E.....Z....~}.|
001ff180  74 4e 63 f9 3e b6 66 03  26 5b 32 79 1d ce 11 a8  |tNc.>.f.&[2y....|
001ff190  71 94 f1 0c b2 51 61 5d  56 15 85 e7 ce e6 ac 20  |q....Qa]V...... |
001ff1a0  9e de 95 89 37 a4 98 ed  a3 48 58 78 ee 1f af 67  |....7....HXx...g|
001ff1b0  63 20 55 57 8b c9 3e 8f  85 9d e9 98 d3 b7 19 55  |c UW..>........U|
001ff1c0  3e 09 f0 19 23 56 d1 10  81 26 43 6c bc cb f6 26  |>...#V...&Cl...&|
001ff1d0  f2 57 dd fc 01 94 a1 a9  fa 5c 99 aa a1 ae b8 fd  |.W.......\......|
001ff1e0  d2 13 85 bb 3e 57 0d 28  9a ff 89 5d 5f 48 a6 e5  |....>W.(...]_H..|
001ff1f0  ab e9 47 78 79 02 1e fc  8e 19 23 b2 a2 fd 86 f5  |..Gxy.....#.....|
001ff200
frostschutz wrote:

if the luks header is damaged that might have happened at any point while the machine was running... anything come to mind? unlikely for pacman to do it

Going back through the journal shows that my fstrim.service timer was run shortly before I rebooted. This was the first and only TRIM operation run after upgrading to a 5.1 kernel which seems to indicate I've hit the bug @Bevan referenced.

May 20 10:07:32 thinkpad systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
May 20 10:07:44 thinkpad fstrim[588]: /home: 146.2 GiB (156943437824 bytes) trimmed on /dev/mapper/home
May 20 10:07:44 thinkpad fstrim[588]: /boot: 344.1 MiB (360796160 bytes) trimmed on /dev/sda1
May 20 10:07:44 thinkpad fstrim[588]: /: 9.7 GiB (10376126464 bytes) trimmed on /dev/mapper/root
May 20 10:07:44 thinkpad systemd[1]: fstrim.service: Succeeded.
May 20 10:07:44 thinkpad systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.

Offline

#6 2019-05-21 17:01:39

frostschutz
Member
Registered: 2013-11-15
Posts: 1,471

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

:-( :-( :-(

operation successful, patient dead, rien ne vas plus

I'm looking for a backup LUKS header

well if you find one, that would change things as non-zero data would be recoverable then. but the filesystem is likely toast anyway, so it would likely reduce you to photorec and recovering unfragmented stuff.

however if TRIM bugged out then it likely will have damaged things everywhere, not just at the start of the partition. in fact you even have to check if the rootfs is still all okay. checksum all the files.

Last edited by frostschutz (2019-05-21 17:06:26)

Offline

#7 2019-05-21 17:06:34

Bevan
Member
Registered: 2009-09-08
Posts: 100

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

I'm sorry to hear that. I'm not sure how long it will take to get this solved upstream. Maybe it's time to open a bug report for Arch to at least get it solved in the Arch linux package.

Offline

#8 2019-05-21 17:08:38

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

Bevan wrote:

Is that volume on an SSD and did you use the "discard" flag in your crypttab?

Yes.
/etc/crypttab

home		UUID=04ed9a68-e7ab-4b8d-82a8-449ea5dbc4bd	/etc/luks-keys/home.luks	discard

smartctl --info /dev/sda

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     SAMSUNG MZ7LN256HCHP-000L7
Serial Number:    S20HNXBG772098
LU WWN Device Id: 5 002538 d00000000
Firmware Version: EMT03L6Q
User Capacity:    256,060,514,304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue May 21 08:59:15 2019 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Bevan wrote:

We are currently hunting an issue with trim/discard on encrypted volumes in Linux 5.1 which may have caused the destruction: https://www.redhat.com/archives/dm-deve … 00084.html

Ouch indeed. sad
From that thread it looks like the bad commit has been established and that kernels >= 5.1 are affected, but in case it helps, my storage setup is as follows:

ext4
dm-crypt (LUKS)
LVM logical volume
LVM single physical volume
GPT partition
SAMSUNG MZ7LN256HCHP-000L7 (OEM SSDs for Lenovo Thinkpad)

Is there any additional information I could provide that might help before I rebuild/restore my home volume?

Offline

#9 2019-05-21 17:09:29

progandy
Member
Registered: 2012-05-17
Posts: 5,259

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

Anyone writing a bug report for the arch kernel? That may be even worth an entry on the frontpage.

Edit: Too slow.

Last edited by progandy (2019-05-21 17:10:10)

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#10 2019-05-21 17:15:09

Bevan
Member
Registered: 2009-09-08
Posts: 100

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

chr0mag wrote:

Is there any additional information I could provide that might help before I rebuild/restore my home volume?

It basically confirms other known cases: A combination of dm-crypt and LVM on Samsung SSDs. Which parts of this play a role and which don't isn't really known so far. I think there's no need to keep the broken volume. I can easilly reproduce the issue here.

Offline

#11 2019-05-21 17:17:58

Bevan
Member
Registered: 2009-09-08
Posts: 100

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

progandy wrote:

Anyone writing a bug report for the arch kernel? That may be even worth an entry on the frontpage.

I'll do. For the meantime, all we can do is spreading the word: https://twitter.com/michael__lass/statu … 2471427072

Offline

#12 2019-05-21 17:20:37

frostschutz
Member
Registered: 2013-11-15
Posts: 1,471

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

Bevan wrote:

I'll do.

Thanks! Also for your bisecting efforts.

Offline

#13 2019-05-21 17:44:24

Bevan
Member
Registered: 2009-09-08
Posts: 100

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

https://bugs.archlinux.org/task/62693

Offline

#14 2019-05-21 17:55:55

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

frostschutz wrote:

:-( :-( :-(

operation successful, patient dead, rien ne vas plus

I'm looking for a backup LUKS header

well if you find one, that would change things as non-zero data would be recoverable then. but the filesystem is likely toast anyway, so it would likely reduce you to photorec and recovering unfragmented stuff.

I don't have a backup LUKS header, but as you say, it seems unlikely this would have helped.

frostschutz wrote:

however if TRIM bugged out then it likely will have damaged things everywhere, not just at the start of the partition. in fact you even have to check if the rootfs is still all okay. checksum all the files.

I would need have known good checksums to compare these with though and I don't run any IDS software on my laptop. The rootfs decrypts & mounts fine, at least up until the point where the home volume can't be found...

Offline

#15 2019-05-21 17:57:34

frostschutz
Member
Registered: 2013-11-15
Posts: 1,471

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

it should be possible to somehow compare installed files to what's in the packages but I forget

oh hang on I think this was it: https://wiki.archlinux.org/index.php/Pa … m_packages

if that comes up empty, either lucky or maybe your rootfs doesn't support trim? but it seemed like it did.

maybe the bug triggers rarely...? so unlucky, and to have killed the LUKS header of all things too, that's at least 2MiB offset off target, 16MiB in case of LUKS2 ;^(

Last edited by frostschutz (2019-05-21 18:04:00)

Offline

#16 2019-05-21 21:15:54

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

frostschutz wrote:

it should be possible to somehow compare installed files to what's in the packages but I forget

oh hang on I think this was it: https://wiki.archlinux.org/index.php/Pa … m_packages

if that comes up empty, either lucky or maybe your rootfs doesn't support trim? but it seemed like it did.

maybe the bug triggers rarely...? so unlucky, and to have killed the LUKS header of all things too, that's at least 2MiB offset off target, 16MiB in case of LUKS2 ;^(

paccheck --md5sum --quiet shows:

linux: '/usr/lib/modules/5.1.3-arch1-1-ARCH/modules.dep' md5sum mismatch (expected f7b3a072fdce73fe206b7937c81a6eb4)
linux: '/usr/lib/modules/5.1.3-arch1-1-ARCH/modules.dep.bin' md5sum mismatch (expected dd5a76ea1fe998b188f7714cb580ff01)
linux: '/usr/lib/modules/5.1.3-arch1-1-ARCH/modules.symbols' md5sum mismatch (expected 9b00551bfb006a53a0f8e37bef6057f1)
linux: '/usr/lib/modules/5.1.3-arch1-1-ARCH/modules.symbols.bin' md5sum mismatch (expected 3a7721aaa438612a9458e046b2db528f)

Reinstalling the package (including re-downloading) doesn't change this. However, it seems this is caused by virtualbox -- specifically virtualbox-host-modules-arch. When I uninstall these paccheck exits cleanly (and when I re-install the same issue reappears). I suspect this is unrelated to my data corruption issue and would be interested if anyone else sees the same thing?

It seems my rootfs volume has emerged in tact, although paccheck wouldn't reveal corrupt files not owned by packages so there's still a small chance of some lingering bad rootfs data.

Thanks @frostschutz & @Bevan for the help.

Offline

#17 2019-05-22 07:38:30

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 23,187

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

That's not a bug and to be expected, a mismatch here will happen as soon as you install out of tree modules. Nothing to worry about.

Offline

#18 2019-05-22 18:03:32

chr0mag
Member
From: Vancouver, Canada
Registered: 2017-02-02
Posts: 94

Re: [SOLVED] Corrupted LUKS volume after upgrade + reboot

Got it. Makes sense, thanks.

Offline

Pages: 1

Board footer

Powered by FluxBB