Arch Linux

destate9

Member

Registered: 2023-09-10

Posts: 8

LUKS home partition fails to deactivate at shutdown

Hello all! Happy to be here as a brand new Arch bro as of this week! Did I mention I use Arch? I started with Ubuntu as my daily driver a few years back and now I'm making the switch.

I'm trying to set up a separate luks encrypted partition for a large chunk of my home directory. Long story short, I'm planning on having multiple distros on this computer and I'd like to have a common home directory. Various posts scared me away from using a shared partition for a shared home directory, so I'm trying to mount a partition that has stuff like "Documents", "Pictures", etc. (not /etc, etc... haha Linux joke), then link those to the respective home directory folders. Anyway, the "why" is not really point of this post (but I'd be happy to get into that too!), rather I have a specific question related to the "how".

I was able to get this scheme to work (albeit with an error at shutdown) by doing the following after setting up my luks partition (with a keyfile) and ext4 filesystem:

Step 1: Modified /etc/crypttab by adding the following line:

`homePart       UUID=[MY UUID]    [PATH TO MY KEYFILE]    luks`

Step 2: Modified /etc/fstab by adding the following line:

`/dev/mapper/homePart                /home/[MY USERNAME]/homePartition ext4 defaults    0 0`

Step 3: Updated the initramfs:

`mkinitcpio -p linux`

Everything worked great! I'd prefer if a mounted drive didn't show up on my Desktop and dock (I'm using Gnome DE), but that's no biggie. I'm trying to troubleshoot the following errors from my journal from shutdown:

Sep 10 10:09:52 [MY HOSTNAME] systemd-cryptsetup[18510]: Device homePart is still in use.
Sep 10 10:09:52 [MY HOSTNAME] systemd-cryptsetup[18510]: Failed to deactivate 'homePart': Device or resource busy

Two seconds later in the log:

Sep 10 10:09:54 [MY HOSTNAME] systemd[1]: Unmounting /home/[MY USERNAME]/homePartition...

Looks like cryptsetup tries to close the luks partition before systemd tries to unmount. How do I ensure that unmounting happens before luksClose? Did I do something wrong with the setup?

destate9

Member

Registered: 2023-09-10

Posts: 8

Re: LUKS home partition fails to deactivate at shutdown

Is there anything additional I need to post? Any additional logs that are needed to diagnose this?

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

Hi, welcome to Arch,
the log excerpts you post don't read bad (because the unmount happens), but it's difficult to say without a full journal filtered for the homePart. What often happens is that some application has not finished unwinding its task when systemd tries to shut the mount down, systemd will retry and all that should be seen in the journal. So you should be able to observe it went well in the end and if unsure, post the full log filtered for the homePart.

Then, also show the hook array you use in mkinitcpio.conf.

One thing for you to try: logout first and powerdown from gdm (if you use it. Or login again to poweroff, if you don't). This way any user files will be done, then observe the log again e.g. with

journalctl -b-1 -u *.mount

What you should definetely change is to enable fsck in fstab for your homePart!

As a sidenote, be wary that while it may not be a big deal to use the same /home for different installations, applications on different releases may also create a mess. As a simple example, think about you upgraded firefox in Arch, but another distro then opens the upgraded profile with the last release. It may work, or work twice and then fail. You can avoid a lot of such problems, if you use different /home/users and link in the data directories you need to work on from both. But that better be a separate topic.

destate9

Member

Registered: 2023-09-10

Posts: 8

Re: LUKS home partition fails to deactivate at shutdown

Thanks for the response!

Here is a more complete log filtered for "homePart" (journalctl -b-1 -g homePart):

Sep 12 15:51:07 [MY_HOSTNAME] systemd[1]: Starting Cryptography Setup for homePart...
Sep 12 15:51:11 [MY_HOSTNAME] systemd[1]: Found device /dev/mapper/homePart.
Sep 12 15:51:11 [MY_HOSTNAME] systemd[1]: Finished Cryptography Setup for homePart.
Sep 12 15:51:11 [MY_HOSTNAME] systemd[1]: Reached target Block Device Preparation for /dev/mapper/homePart.
Sep 12 15:51:11 [MY_HOSTNAME] systemd[1]: Mounting /home/[MY_USERNAME]/homePartition...
Sep 12 15:51:11 [MY_HOSTNAME] systemd[1]: Mounted /home/[MY_USERNAME]/homePartition.
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: Stopping Cryptography Setup for homePart...
Sep 12 15:53:45 [MY_HOSTNAME] systemd-cryptsetup[3161]: Device homePart is still in use.
Sep 12 15:53:45 [MY_HOSTNAME] systemd-cryptsetup[3161]: Failed to deactivate 'homePart': Device or resource busy
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: systemd-cryptsetup@homePart.service: Control process exited, code=exited, status=1/FAILURE
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: systemd-cryptsetup@homePart.service: Failed with result 'exit-code'.
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: Stopped Cryptography Setup for homePart.
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: Unmounting /home/[MY_USERNAME]/homePartition...
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: home-[MY_USERNAME]-homePartition.mount: Deactivated successfully.
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: Unmounted /home/[MY_USERNAME]/homePartition.
Sep 12 15:53:45 [MY_HOSTNAME] systemd[1]: Stopped target Block Device Preparation for /dev/mapper/homePart.

The hook array I use in mkinitcpio.conf is: HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block encrypt filesystems fsck)

I tried logging out and then shutting down. I still saw the same behavior. I haven't gone any further than just mounting this drive to a new directory (which isn't home yet), so nothing should be using this directory yet.

Great tip with the fsck! I changed that from 0 to 2. That is the correct choice, right?

I agree on your note about the shared home partition! I plan on experimenting with some small portions of my home directory shared (like Documents, Pictures, etc.).

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

Ok, yes "2" is right. From your log excerpt we now see that it's happening at once (same second) this time, and since homePart is empty it can't be due to open files. Perhaps it happens, because it is nested into /home (which you probably have as another separate device..), but the order of the systemd-cryptsetup unit called first on shutdown sure looks wrong. If you look at systemd.mount(5), one can add dependency flags in fstab (x-systemd.xyz) to influence behaviour of the automount generator, but it would need a bit more diving into your setup.
Generally, I'd say the error is nothing to worry about, the luks header is static as long as you don't operate commands on it, i.e. the important part for the data is that the unmount operation is successful. And if your /home partition is not unmounted cleanly, you'd see another error, that would be serious. To investigate further, you better observe the full shutdown log for the mounts and for the units, perhaps that shows more.
I can tell you that I have a populated crypttab for devices mounted to /srv/NAS and /mnt/homePart alikes, and logs don't show such a race condition when systemd shuts them down with default autogenerated units. An option you have is to use an overlay mount to link into home too.

destate9

Member

Registered: 2023-09-10

Posts: 8

Re: LUKS home partition fails to deactivate at shutdown

For some reason now the unmount attempt and the cryptsetup operationn are happening in the same second, but in earlier boots it was separated by a few seconds. Either way, it seems to be in the wrong order.

You may be on to something with the nesting being part of the problem. My other two partitions I'm working with for this system is an unencrypted /boot partition and a luks encrypted root partition. I'm mounting the partition that I'm having trouble with at /home/[MY_USERNAME]/homePartition, but with "/" being a separate luks partition, maybe something is holding up the unmounting of the "homePartition".

For completeness, here is my full /etc/fstab file:

/dev/mapper/root       /                                    ext4    errors=remount-ro    0    1
UUID=[MY_UUID]         /boot                                ext4    rw,relatime          0    2
/dev/mapper/homePart   /home/[MY_USERNAME]/homePartition    ext4    defaults             0    2

And again, my /etc/crypttab file only has the one line:

homePart    UUID=[MY_HOMEPART_UUID]    /home/george/homePartLuks.key    luks

I'd gladly go with a x-systemd.requires or x-systemd.required-by option to make this work, but alas I am too much of a n00b to know some of the basics around how those requirements work. I would think that I would want to tell systemd that the mount unit requires the cryptsetup unit (which I thought was dynamically created so can I even reference that unit?), but would that be enough to tell systemd to tear down the mount unit before the cryptsetup unit?

A few hacky-looking solutions I've found here and there on the interwebs mention doing something where I write an extra unit that does the unmount, then somehow invoke that unit as closing dependency of cryptsetup, but this seems like an odd solution as I'm somewhat manually unmounting the automounted drive. It just doesn't seem like that should be necessary.

Thanks for the heads up on the overlay mount setup! That sounds almost exactly like what I want and I had no idea that existed! I'll have to chew on that a little longer to see if that's going to work. In either case, I think I'd still end up with the same weird unmount vs. encrypt race, so I'd still like to get to the bottom of that.

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

It sure is the wrong approach to try mix the systemd autogenerator with custom mounts, at least nowadays. Either rely on the generator and its supported fstab options for a mount, or create custom units for a specific mount and ensure the autogenerator ignores it. If you want to experiment finding the right fstab options: yes, dynamic creation does not mean the naming is dynamic (look at systemd-cryptsetup@homePart.service).

Before you try working with overlays, another simpler option to test around the nested situation is to just change your existing fstab to mount the device to /mnt or /mnt/homePart, create a symlink like "ln -s /mnt/homePart /home/[username]/homePart" and check if that gives you the same error after a reboot/shutdown.

destate9

Member

Registered: 2023-09-10

Posts: 8

Re: LUKS home partition fails to deactivate at shutdown

Good thought with the nested mount situation, but I'm still running into the same issue if I mount homePart at /mnt/homePartition instead of /home/[MY_USERNAME]/homePartition unfortunately.

To me it doesn't seem like there are too many variables to mess up here. Besides /etc/fstab and /etc/crypttab, what else could I have messed up?

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

Nothing. I'm now back at the desktop with said crypttab mounts and realize I must have misfiltered my own journal, when I made above claim that I don't have similar failures (sorry). They actually show up pretty consistently : )
Yet, as I wrote above it's nothing to worry about IMO, since the clean unmount is what counts. When I get to look into it and find something soon, I can update here but I would not invest much into it since tinkering with conditions can slow down a shutdown considerably.

destate9

Member

Registered: 2023-09-10

Posts: 8

Re: LUKS home partition fails to deactivate at shutdown

Well I'm very happy to learn that I'm not the only one! I'll table this for now and save it for a rainy day when I'm really bored and hopefully more read up on how systemd units work so I can more effectively troubleshoot. Thanks for all the help!

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

We're definetely not the only ones with the problem. I did a search for the issue and arrived at a number of systemd issues for it.
During your rainy day follow-up, have a look at https://github.com/systemd/systemd/issues/8472
It gives a good overview about systemd units and testing steps.
Later on it links to 1620 which appears to be the original issue and is unresolved since 2015 ...
I also tried a systemd debug boot and it shows another last line on shutdown for the device mapper (i.e. after the unmount), which may be when systemd finally disassembles device leftovers.

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

I've now tried a suggested workaround from the bug reports (ref #1620 comment) that seems to solve the shutdown order without noticeable delay.

For you it would be to add

x-systemd.requires=systemd-cryptsetup@homePart.service

to the fstab options of the /dev/mapper/homePart partition.

mattshahn

Member

Registered: 2023-09-26

Posts: 1

Re: LUKS home partition fails to deactivate at shutdown

I don't think OP is doing anything wrong here. I started noticing these errors in my system journal after upgrading systemd from 253 to 254. I have opened a bug on the systemd Github.

https://github.com/systemd/systemd/issues/29336

tekstryder

Member

Registered: 2013-02-14

Posts: 179

Re: LUKS home partition fails to deactivate at shutdown

Fortunately fixed here:
https://github.com/systemd/systemd/pull/29338

Thanks for filing @mattshahn!

Hopefully lands in 254 stable.

Last edited by tekstryder (2023-09-27 11:41:52)

destate9

Member

Registered: 2023-09-10

Posts: 8

Re: LUKS home partition fails to deactivate at shutdown

Thanks so much everyone! Looks like the PR is slated for 255? Either way, I made the tweak on my local machine and it works! I'm glad there's not only a solution, but also that I can see exactly what needed to be changed. I learned some systemd today... it was a good day.

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,487

Re: LUKS home partition fails to deactivate at shutdown

Yes, 255. I've removed the manual x-systemd.requires= from fstab and some reboots later the luks device continues to unmount cleanly. So, it indeed seems to be solved upstream, yay.