You are not logged in.

#1 2020-01-04 00:27:04

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

[SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I'm not sure what happened, I was using cryptsetup to encrypt my filesystem for a few years now. Suddenly new system I installed today won't boot if I have crypt swap uncommented in my fstab...

I'm probably missing something simple - or - something fundamentally changed since last time I installed system with swap.

This is my crypttab entry

swap      /dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:1:2    /dev/urandom   swap,cipher=aes-cbc-essiv:sha256,size=256

This is my fstab:

/dev/mapper/swap                          none  swap defaults        0 0

I never comment out crypttab entry - when I manage to boot system with fstab commented I can see `/dev/mapper/swap` and also I can run `swapon /dev/mapper/swap` resulting in swapon --show showing swap as expected...

I have done some googling and people suggest systemd might kick in with some generated units. To be honest I'm not that deep with systemd, but I would not want it to do stuff for me automatically, it's not windows right? Anyways my other systems with similar setup have no problem booting and I keep them up to date. So I must be missing something silly... Please help...

Last edited by Gregosky (2020-01-17 08:04:06)

Offline

#2 2020-01-05 02:37:13

cmurf
Member
Registered: 2019-09-06
Posts: 27

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I've seen a similar problem starting 9 months ago on Fedora, but I haven't tried it lately. But maybe this will give you something to go on as you troubleshoot it.
https://bugzilla.redhat.com/show_bug.cgi?id=1691589

My speculation is some kind of race or even something related to randomness, because I too have this working on one system but not another system, with all the same versions of things. But the CPUs are different and how randomness gets initially seeded maybe different and that's why it gets stuck? Thing is the cryptsetup does succeed. The hang for me is somewhere in between mkswap and swapon.

Last edited by cmurf (2020-01-05 03:05:28)

Offline

#3 2020-01-05 12:15:07

loqs
Member
Registered: 2014-03-06
Posts: 17,197

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

Offline

#4 2020-01-05 18:55:17

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

@cmurf, @loqs - many thanks for your responses.

On my side I noticed that as soon as I do this:

systemctl disable systemd-networkd

and reboot... everything works fine (except, of course, network).

I can do this:

swapon --show

and I get my swap listed. However as soon as I run `systemctl enable systemd-networkd` and reboot - system hangs on waiting for swap...

I have no idea what does systemd-networkd have to do with swap.

Offline

#5 2020-01-05 20:56:25

loqs
Member
Registered: 2014-03-06
Posts: 17,197

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

If you randomly press keys does boot then proceed?

Offline

#6 2020-01-05 21:22:04

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

Hi @loqs,

I tried to press keys randomly to no avail. This is what I see at boot time:

A password is required to access the root volume:
Enter passphrase for /dev/sdc1:
/dev/mapper/root: clean, 46886/1499136 files, 581518/5992192 blocks
[FAILED] Failed to start Network Service.
See 'systemctl status systemd-networkd.service' for details.
[  OK  ] Stopped Network Service.
Starting Network Service...
[  OK  ] Started Cryptography Setup for swap.
[  OK  ] Reached target Local Encrypted Volumes.
[FAILED] Failed to start Network Service.
See 'systemctl status systemd-networkd.service' for details.
[  OK  ] Stopped Network Service.
Starting Network Service...
[FAILED] Failed to start Network Service.
See 'systemctl status systemd-networkd.service' for details.
[  OK  ] Stopped Network Service.
Starting Network Service...
[FAILED] Failed to start Network Service.
See 'systemctl status systemd-networkd.service' for details.
[  OK  ] Stopped Network Service.
Starting Network Service...
[FAILED] Failed to start Network Service.
See 'systemctl status systemd-networkd.service' for details.
[  OK  ] Stopped Network Service.
[FAILED] Failed to start Network Service.
See 'systemctl status systemd-networkd.service' for details.
[     *] A start job is running for /dev/mapper/swap (3min 24s / no limit)

And then it hangs for ever...

I thought that maybe due to small amount of memory my build process failed somewhere so I added swap at build time and redone everything from scratch. I still see the same problem - if I have fresh system and I add swap using crypttab and fstab then rebooting works fine and everything seems to work consistently (I was rebooting number of times to ensure it's not a glitch).

But as soon as I add systemd-networkd to the mix I get the above output.

Offline

#7 2020-01-05 21:23:12

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

Gregosky wrote:
swap      /dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:1:2    /dev/urandom   swap,cipher=aes-cbc-essiv:sha256,size=256

What is /dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:1:2 for you? Does it still exist? Just asking since sometimes, changing bios settings can change these, too.

I strongly dislike this kind of swap specification. You're basically asking for one of your raw devices to be formatted here. If you pick the wrong device, it can lead to disastrous data loss.

Tried to work around it with https://wiki.archlinux.org/index.php/Dm … _and_LABEL (basically a hack to add some sanity checking) but it's still not great.

Edit: didn't see your latest post, might be something else entirely

Last edited by frostschutz (2020-01-05 21:24:45)

Offline

#8 2020-01-06 05:20:04

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

Thanks @frostschutz - The system I'm trying to make work is running on Linode. On their GUI you can simply select block device to appear as `/dev/sdX`. I think you are right that placing anything disk/by-path in crypttab brings nothing compared with simple naming, not sure what was I thinking when I was doing this, it was late at night...

Anyways - as you also noticed swap works fine as long as I don't enable systemd-networkd. Weird.

Which means the issue is there and I have no idea how to approach it. Systemd makes my system not bootable if I enable systemd-networkd service. I was thinking about trying to reduce systemd-netwporkd priority to lowest possible (or to add artificial swap dependency).

Last edited by Gregosky (2020-01-06 20:48:36)

Offline

#9 2020-01-09 21:10:04

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I configured system with normal (unencrypted) swap and with systemd-networkd and boot process is not hanging.

Then I added dependency for systemd-networkd to start after swap.target and this is resulting in boot not hanging

systemd-networkd shows as failed once on the beginning with the "Could not set hostname: Permission denied" error but this is separate issue.

I'd love to understand why is systemd-networkd in conflict with encrypted swap.

---
Edit

I rebooted once - everything worked fine (apart from error mentioned above)
Then I added systemd-resolved and rebooted - and again boot process hanged like before...
So I reverted this change (systemctl disable systemd-resolved) but I'm back to square one. I thought adding dependency to swap.target resolved the issue - it did not.

Last edited by Gregosky (2020-01-09 21:37:48)

Offline

#10 2020-01-10 07:21:51

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I managed to make swap timeout during boot (rather than hold it for ever) by modifying swap entry in fstab like this:

/dev/mapper/swap none swap sw,nofail 0 0

and crypttab like this:

swap /dev/sdb1 /dev/urandom swap,nofail,cipher=aes-cbc-essiv:sha256,size=256,timeout=10

And then I was able to check logs.

What drew my attention is that:
* systemd-networkd fails to start because of dbus athentication issues
* dbus starts much later than systemd-networkd tries to use it
* crypt device for swap is timing out however when system finally boots I can see /dev/mapper/swap

If I disable systemd-networkd system boots without errors. So I am trying to understand why is systemd-networkd breaking my boot sad

Last edited by Gregosky (2020-01-10 07:28:45)

Offline

#11 2020-01-10 07:25:35

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I enabled haveged and now I have boots consistently not failing. So the answer is there was not enough entropy... I'm stupid.

Offline

#12 2020-01-12 19:09:58

cmurf
Member
Registered: 2019-09-06
Posts: 27

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I don't have systemd-networkd on any of my systems, so that definitely isn't related to the failures I was seeing.

Offline

#13 2020-01-13 19:22:46

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

@cmurf - thanks, as I figured the failure was not really related to systemd-networkd, but lack of entropy to start crypto-swap (I guess systemd-networkd was just catalyst since it probably was drawing some of entropy so there was not enough left for crypto-swap).

Offline

#14 2020-01-14 20:04:18

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I was happy to see my boot not to hang, then a few boots later the problem is back again.

I come to think what I'm seeing is what @cmurf linked to - some sort of race condition between cryptsetup/mkswap/swapon

In essence without changing anything - only rebooting machine - every 3rd/4th boot succeeds. It feels very random and I guess the fact that I'm building on nanode (1CPU, 1Gb RAM) is somehow contributing to this issue.

It is strange that service for crypto-swap starts and then boot hangs on swap device...

Starting Cryptography Setup for cryptoswapsdb...
[  OK  ] Started Cryptography Setup for cryptoswapsdb.
(...)
[ TIME ] Timed out waiting for device /dev/mapper/cryptoswapsdb.
[DEPEND] Dependency failed for /dev/mapper/cryptoswapsdb.
[DEPEND] Dependency failed for Swap.

Also if I add `noauto` to fstab entry system boots OK and I can see crypt-swap device is there. I can enable swap manually (I don't even have to run mkswap on it)

Adding `x-systemd.device-timeout=10s` to both fstab and crypttab helped to reduce time between boots, as I am rebooting this machine a lot experimenting with whatever setting makes sense (https://www.freedesktop.org/software/sy … mount.html , https://www.freedesktop.org/software/sy … .swap.html , https://www.freedesktop.org/software/sy … ttab.html)

Experimenting with ciphers led me to have less failing boots (weird...), this one seems to give good results

cipher=aes-xts-plain64,size=256

But it can be also that neighbors on the host I'm testing on are just less active (or something else, maybe solar wind)

Yup, it was solar wind. After two consecutive successful boots third one failed (although host felt much less responsive and it took much longer to decrypt root partition)

Last edited by Gregosky (2020-01-15 07:02:25)

Offline

#15 2020-01-15 09:28:29

Gregosky
Member
From: UK
Registered: 2013-07-26
Posts: 173

Re: [SOLVED] Boot hangs when dm-crypt swap and systemd-networkd is enabled

I was experimenting with systemd-networkd boot order and figured out acceptable option is to just add startup delay to systemd-networkd (or start systemd-networkd with timer using OnBootSec option)

Timer like below works for me:

[Unit]
Description=Delaying systemd-networkd start

[Timer]
OnBootSec=60sec
Unit=systemd-networkd.service
Unit=systemd-resolved.service

[Install]
WantedBy=multi-user.target

Once created (I dropped this file under /etc/systemd/system) do the following:

systemctl disable systemd-networkd
systemctl disable systemd-resolved
systemctl enable systemd-networkd.timer

Important bit is to have x-systemd.device-timeout option set in both fstab and crypttab.

My fstab:

/dev/mapper/cryptoswapsdb none swap sw,x-systemd.device-timeout=10 0 0

My crypttab:

cryptoswapsdb /dev/sdb1 /dev/urandom swap,cipher=aes-xts-plain64,size=256,x-systemd.device-timeout=10s

Now I get all boots with network. Some boots fail on swap setup but at least I always have network. The price I pay is to wait some time before network is available.

--------

There is better solution (mentioned here: https://github.com/systemd/systemd/issu … 425743011)

Simply create file like below within /etc/systemd/system/systemd-cryptsetup@cryptoswapsdb.service.d/90-trigger-udev.conf (your service name will be different, take it from crypttab)

# Run udevadm trigger after the mkswap call in the original generated
# service

[Service]
ExecStartPost=/sbin/udevadm trigger /dev/mapper/%i

With this in place there is no need for noauto and nor for timer.

Last edited by Gregosky (2020-01-17 08:03:38)

Offline

Board footer

Powered by FluxBB