You are not logged in.

#1 2019-10-15 07:04:14

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

[solved] Issue with file system after hibernation

Hi,

On a ThinkPad T480 I have Arch running. Except for an issue with the file system being corrupt after the machine resumes from hibernation it works perfectly.
I spent some time searching for a solution but can't come up with one. Therefore I'm hoping to find help here.

The setup is as follows: There still is a partition with the former Windows installation, which is never used though. Still, I made sure fast boot is disabled there. Anyway, this has last been booted like 2 months ago and if this turns out to be an issue, I'd just get rid of the Windows partition.
Regarding Arch, the the setup is that it is installed on a luks encrypted ext4 partition. I tried btrfs before, but that had the same problem. Ext4 seems to be able to recover from the issue on fsck at reboot, btrfs couldn't and was damaged beyond repair after hibernating. Which is why I reinstalled with Ext4 then.

Swap is a seperate partition a little larger than the RAM, also encrypted with luks. I use openswap for this.

Here is what I find in the logs after hibernation:

Okt 15 08:40:17 xyz kernel: EXT4-fs (dm-0): Delayed block allocation failed for inode 16515321 at logical offset 355 with max blocks 3 with error 117
Okt 15 08:40:17 xyz kernel: EXT4-fs (dm-0): This should not happen!! Data will be lost
Okt 15 08:40:17 xyz kernel: EXT4-fs error (device dm-0): ext4_validate_inode_bitmap:99: comm Cache2 I/O: Corrupt inode bitmap - block_group = 2002, inode_bitmap = 65536018
Okt 15 08:40:17 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_inode:355: Filesystem failed CRC
Okt 15 08:40:17 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_inode:355: Corrupt filesystem

Okt 15 08:33:19 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_blocks:4952: Filesystem failed CRC
Okt 15 08:33:19 xyz kernel: EXT4-fs error (device dm-0): ext4_validate_block_bitmap:376: comm Thread (pooled): bg 2241: bad block bitmap checksum

Okt 15 08:32:43 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_inode:355: Corrupt filesystem
Okt 15 08:32:43 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_inode:355: Corrupt filesystem
Okt 15 08:32:43 xyz kernel: EXT4-fs error (device dm-0): ext4_validate_inode_bitmap:99: comm Timer-0: Corrupt inode bitmap - block_group = 1984, inode_bitmap = 65011728


Okt 15 08:30:30 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_inode:355: Corrupt filesystem

Okt 15 08:30:05 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_inode:355: Corrupt filesystem

Okt 15 08:28:53 xyz kernel: EXT4-fs error (device dm-0): ext4_validate_block_bitmap:376: comm logrotate: bg 772: bad block bitmap checksum
Okt 15 08:28:53 xyz kernel: EXT4-fs error (device dm-0) in ext4_free_blocks:4952: Filesystem failed CRC

When it says data will be lost I usually do a regular reboot. This brings up fsck which fixes the corrupt file system very quickly and the errors do not occur again until I hibernate the machine again. When booting normally after shut down instead of hibernate the errors do not come up. Also suspend to ram works flawlessly and does not provoke the problem.

My hooks line:

HOOKS=(base udev autodetect keymap keyboard modconf block encrypt openswap filesystems fsck resume)

Also had the resume hook after openswap originally and moved it in order to tackle the problem, but it had no effect at all.

Here are the kernel parameters:

GRUB_CMDLINE_LINUX_DEFAULT="resume=UUID=995511b4-be63-401c-a620-08eaab78b677 modprobe.blacklist=nouveau loglevel=3 scsi_mod.scan=sync"
GRUB_CMDLINE_LINUX="cryptdevice=UUID=1faa404c-7a01-4361-a459-20be8243ac73:sysroot"

The scsi_mod.scan option is also something I came up with while trying to fix the issue, but apparently also has no effect.

Apart from the log messages I notice no ill behavior after resuming from hibernation.

I hope you can help me out here, as I'd really like to make use of hibernation. My workaround so far is to not hibernate.
If I'm missing to post some important information here, please let me know and I'll provide that of course.

Thanks!


Summary for future readers with the same or a similar problem, stumbling here by some search engine:

  • File system on encrypted partition got corrupted after wake up from hibernation (suspend to disk)

  • Reason was that the key for unencrypting the swap partition was read from the encrypted root file system

  • In order to do so, the root partition was swiftly mounted to read the key file and then unmounted again

  • This intermediate mount while the root file system was not yet back from hibernation corrupted the file system

  • Solution/workaround is to not provide the key for the swap partition on the root partition. In my case I switched to a swap file instead of separate partition, because the swap file gets encrypted/decrypted along with the root file system without extra steps

Last edited by gerald82 (2019-10-24 19:59:47)

Offline

#2 2019-10-15 10:45:43

sabroad
Member
Registered: 2015-05-24
Posts: 224

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

My hooks line:

HOOKS=(base udev autodetect keymap keyboard modconf block encrypt openswap filesystems fsck resume)

Put resume before filesystems/fsck, otherwise they'll be mounted and "fixed" before resuming.

https://wiki.archlinux.org/index.php/Dm … tcpio_hook:

Add the hook openswap in the HOOKS array in /etc/mkinitcpio.conf, before filesystem but after encrypt. Do not forget to add the resume hook after openswap.

HOOKS=(... encrypt openswap resume filesystems ...)

--
saint_abroad

Offline

#3 2019-10-16 05:30:51

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

sabroad wrote:

Put resume before filesystems/fsck, otherwise they'll be mounted and "fixed" before resuming.

https://wiki.archlinux.org/index.php/Dm … tcpio_hook:

Add the hook openswap in the HOOKS array in /etc/mkinitcpio.conf, before filesystem but after encrypt. Do not forget to add the resume hook after openswap.

HOOKS=(... encrypt openswap resume filesystems ...)

Thanks for the reply.
That is exactly what it was like when I first noticed the problem.
I gave it another shot and the HOOKS array is now this:

HOOKS=(base udev autodetect keymap keyboard modconf block encrypt openswap resume filesystems fsck)

As I expected since that was the original line, it didn't help. I did not forget to run mkinitcpio after the change. The problem still occurs after resuming from hibernation.

Offline

#4 2019-10-16 06:34:38

mxfm
Member
Registered: 2015-10-23
Posts: 163

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

Hi,
The setup is as follows: There still is a partition with the former Windows installation, which is never used though. Still, I made sure fast boot is disabled there. Anyway, this has last been booted like 2 months ago and if this turns out to be an issue, I'd just get rid of the Windows partition.
Regarding Arch, the the setup is that it is installed on a luks encrypted ext4 partition. I tried btrfs before, but that had the same problem. Ext4 seems to be able to recover from the issue on fsck at reboot, btrfs couldn't and was damaged beyond repair after hibernating. Which is why I reinstalled with Ext4 then.

So, basically you somehow misconfigured fstab/crypttab/hooks which corrupted ext4 partition. Then you reformatted it to btrfs partition and your setup corrupted it too. Then you reformatted partition to ext4 and it was corrupted again. It seems you have some error in fstab or boot mount options. Please post fstab/crypttab and blkid.

Offline

#5 2019-10-16 07:21:37

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

mxfm wrote:

So, basically you somehow misconfigured fstab/crypttab/hooks which corrupted ext4 partition. Then you reformatted it to btrfs partition and your setup corrupted it too. Then you reformatted partition to ext4 and it was corrupted again. It seems you have some error in fstab or boot mount options. Please post fstab/crypttab and blkid.

No, the original setup was with btrfs. That probably had the same problem, but completely broke instead of resuming from hibernation. So much so, I wasn't able to recover the file system. It was the first time I tried btrfs and thought maybe it's still not that stable...
I cleanly reinstalled from scratch then. Basically the same setup, except I replaced btrfs by ext4. I just wanted to mention btrfs had a similar or the same problem, but there is nothing left from that installation.

As for my current system:
crypttab is empty/only commented-out lines. I never touched it.
fstab:

# Static information about the filesystems.
# See fstab(5) for details.

UUID=68D6-6020                                  /boot/EFI       vfat            defaults,rw     0 1

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/mapper/sysroot LABEL=SYSROOT
UUID=20b9ac0c-c179-4645-a363-d5cb855836dc       /               ext4            rw,relatime     0 1

UUID=995511b4-be63-401c-a620-08eaab78b677       swap            swap            defaults        0 0

UUID=61620564-5fe8-4c3f-95c1-cb1d01361a10       /opt/VmMount    xfs             noauto,relatime,users,noexec    0 0

And blkid:

/dev/nvme0n1p1: LABEL="SYSTEM" UUID="68D6-6020" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="44454cb1-d7fc-4be6-ac72-93fa731e6b4a"
/dev/nvme0n1p2: PARTLABEL="Microsoft reserved partition" PARTUUID="9e2b2286-edf9-420d-8728-b94522749362"
/dev/nvme0n1p3: TYPE="BitLocker" PARTLABEL="Basic data partition" PARTUUID="88f89e23-4cf1-4530-8cbe-f2abd1786ae1"
/dev/nvme0n1p4: LABEL="WinRE_DRV" UUID="BC3CDF8C3CDF3FD6" TYPE="ntfs" PARTLABEL="Basic data partition" PARTUUID="793b63e1-f534-47d5-a70d-32c9d147058f"
/dev/nvme0n1p5: UUID="1faa404c-7a01-4361-a459-20be8243ac73" TYPE="crypto_LUKS" PARTLABEL="SYSROOT" PARTUUID="3016bff7-956c-433b-93c5-ebe563667ee2"
/dev/nvme0n1p6: UUID="41acb3ef-236e-40b2-b723-b02fac1a886d" TYPE="crypto_LUKS" PARTLABEL="LSWAP" PARTUUID="f21dd2e9-892e-4014-8262-50e94edb5d96"
/dev/sda1: UUID="2e61367a-4361-462c-9e6f-8d70a2aa5339" TYPE="crypto_LUKS" PARTUUID="bbae5776-417a-cd4c-87e8-9d9eb34fa68e"
/dev/mapper/sysroot: LABEL="SYSROOT" UUID="20b9ac0c-c179-4645-a363-d5cb855836dc" TYPE="ext4"
/dev/mapper/lswap: UUID="995511b4-be63-401c-a620-08eaab78b677" TYPE="swap"

nvme0n1p5 is the partition that has the issue.

Offline

#6 2019-10-16 09:03:01

mxfm
Member
Registered: 2015-10-23
Posts: 163

Re: [solved] Issue with file system after hibernation

You have 'resume=UUID=995511b4-be63-401c-a620-08eaab78b677' (encrypted luks) in kernel parameters and 'UUID=995511b4-be63-401c-a620-08eaab78b677' (decrypted swap) in fstab. Since there is no cryptab, who decrypts swap partition?

Offline

#7 2019-10-16 11:01:35

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

mxfm wrote:

You have 'resume=UUID=995511b4-be63-401c-a620-08eaab78b677' (encrypted luks) in kernel parameters and 'UUID=995511b4-be63-401c-a620-08eaab78b677' (decrypted swap) in fstab. Since there is no cryptab, who decrypts swap partition?

OpenSwap does that.
Here's the OpenSwap configuration:

## cryptsetup open $swap_device $crypt_swap_name
## get uuid using e.g. lsblk -f
swap_device=/dev/disk/by-uuid/41acb3ef-236e-40b2-b723-b02fac1a886d
crypt_swap_name=lswap

## one can optionally provide a keyfile device and path on this device
## to the keyfile
keyfile_device=/dev/mapper/sysroot
keyfile_filename=etc/keyfile-lswap

## additional arguments are given to mount for keyfile_device
## has to start with --options (if so desired)
#keyfile_device_mount_options="--options=subvol=__active/__"

## additional arguments are given to cryptsetup
## --allow-discards options is desired in case swap is on SSD partition
cryptsetup_options="--type luks1 --allow-discards"

Offline

#8 2019-10-16 11:24:46

sabroad
Member
Registered: 2015-05-24
Posts: 224

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

Swap is a seperate partition a little larger than the RAM, also encrypted with luks. I use openswap for this.

I'll presume AUR package. What are the contents of /etc/openswap.conf ?


--
saint_abroad

Offline

#9 2019-10-16 11:33:45

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

sabroad wrote:
gerald82 wrote:

Swap is a seperate partition a little larger than the RAM, also encrypted with luks. I use openswap for this.

I'll presume AUR package. What are the contents of /etc/openswap.conf ?

Yes. I posted the config above.
Now, reading through openswap.hook, I realize it mounts the $keyfile_device, which happens to be my sysroot, then reads the $key_file and then unmounts the $keyfile_device again.
I'm not really familiar with what exactly happens on resume after hibernation, but since the openswap hook is after the encrypt hook in my hooks array, at the time when openswap runs, the root file system should already be available, right? If so, it wouldn't need to mount that device in the first place. And this double mount might actually cause the issue.
Now, I don't want to edit the openswap.hook script, but would it help to pass a readonly mount option to it? This seems to be possible in the config.

Offline

#10 2019-10-16 12:18:19

sabroad
Member
Registered: 2015-05-24
Posts: 224

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

Now, reading through openswap.hook, I realize it mounts the $keyfile_device, which happens to be my sysroot, then reads the $key_file and then unmounts the $keyfile_device again.

Not a good idea. The wiki has a warning:

Mounting the file system is dangerous and destructive. The keyfile should not be read from the root file system.

The linked warning states:

* BIG FAT WARNING *********************************************************
*
* If you touch anything on disk between suspend and resume...
*				...kiss your data goodbye.

$keyfile_device should be something other than your root device. !IMPORTANT!

addendum:

gerald82 wrote:

would it help to pass a readonly mount option to it?

Unfortunately not:

Be sure not to mount anything, not even read-only mount, or you are going to lose your data.

Last edited by sabroad (2019-10-16 12:24:41)


--
saint_abroad

Offline

#11 2019-10-17 06:19:03

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

Any suggestions on how to solve this?
I could setup a seperate partition with the key. But when I encrypt this I will either need to type in the password everytime or provide the keyfile another way. Which is exactly the same problem, so no good.
If I provide the keyfile on an unencrypted device, I can just as well not encrypt anything. However, it's my work machine and the employer requires storage encryption.
I'm fine with entering a password on boot once, and then my user password for login of course, but I don't want to end up having to enter 3 or more passwords before being logged in.

The only other solution I could probably live with is using a SD card with an unencrypted partition that contains the keyfile. But then I must never forget to remove the SD card when I leave the machine and also must not forget to plug it in before I hit the power button.

I think using a swap file instead of a seperate swap partition might be the best option. That way it'll be encrypted on the root partition and be available along with that without decrypting seperately, not requiring a seperate key/password. Am I missing something?

Addendum:
The major reason I opted for a swap partition instead of swapfile was that swap file and hibernation might not work with btrfs, but since I no longer use btrfs I could replace the swap partition by a file.

Last edited by gerald82 (2019-10-17 06:22:15)

Offline

#12 2019-10-17 07:06:35

mxfm
Member
Registered: 2015-10-23
Posts: 163

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

Any suggestions on how to solve this?
I could setup a seperate partition with the key. But when I encrypt this I will either need to type in the password everytime or provide the keyfile another way. Which is exactly the same problem, so no good.

You need to type password once at boot, further can be configured with keyfiles, which are placed at encrypted storage. In case of swap, you can use swapfile.


gerald82 wrote:

I think using a swap file instead of a seperate swap partition might be the best option. That way it'll be encrypted on the root partition and be available along with that without decrypting seperately, not requiring a seperate key/password. Am I missing something?

Addendum:
The major reason I opted for a swap partition instead of swapfile was that swap file and hibernation might not work with btrfs, but since I no longer use btrfs I could replace the swap partition by a file.

Btrfs supports swapfiles since kernel 5.0, but you can stick to ext4.

Offline

#13 2019-10-17 10:17:17

sabroad
Member
Registered: 2015-05-24
Posts: 224

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

I think using a swap file instead of a seperate swap partition might be the best option. That way it'll be encrypted on the root partition and be available along with that without decrypting seperately, not requiring a seperate key/password.

That's what I'd recommend as the simple route.

However, if you want to try keeping your partitions, systemd's sd-encrypt hook will cache passwords to unlock multiple devices:

Note: When using systemd-boot and the sd-encrypt hook, if a non-root partition's passphrase is the same as root's, there is no need to put that non-root partition in crypttab due to passphrase caching.

Last edited by sabroad (2019-10-17 10:18:10)


--
saint_abroad

Offline

#14 2019-10-24 19:49:02

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

Hi,

today I implemented the solution with a swap file.
So far it looks good, the file system seems to not get broken any more.

Thank you very much, sabroad and mxfm!

I'm marking the thread solved, albeit frankly it feels a bit like a workaround instead of a real solution.

Offline

#15 2019-10-25 09:27:46

mxfm
Member
Registered: 2015-10-23
Posts: 163

Re: [solved] Issue with file system after hibernation

gerald82 wrote:

Hi,

today I implemented the solution with a swap file.
So far it looks good, the file system seems to not get broken any more.

Thank you very much, sabroad and mxfm!

I'm marking the thread solved, albeit frankly it feels a bit like a workaround instead of a real solution.

Why do you think your solution is a workaround?

Offline

#16 2019-10-29 07:38:49

gerald82
Member
From: The known universe
Registered: 2011-03-25
Posts: 83

Re: [solved] Issue with file system after hibernation

mxfm wrote:

Why do you think your solution is a workaround?

Don't get me wrong, I'm fine with the solution, especially since I originally intended to have a swap file but opted for a partition instead because BTRFS support for swap files was not available or just too brand new back when I set it up.
But somehow I think there should be a way to cleanly do this with a swap partition. Or moreover, I do not understand why mounting the file system read-only while in hibernation to obtain the key file causes such problems. Looks like it is not entirely read-only then.
However, while I'm fine with a swap file, there might be people who clearly prefer a swap partition and for these people there is no simple solution for the problem.

Offline

Board footer

Powered by FluxBB