[SOLVED] curl in initramfs hangs at getrandom

Curly060 · 2018-10-22 20:34:35

Hi,

I have a custom initramfs hook which uses curl to make an HTTPS call. This used to work just fine, but recently (*) curl hangs for a looong time. I have traced this down with strace and the last thing I see before it hangs is this:

openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 4
fstat(4, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
openat(AT_FDCWD, "/dev/random", O_RDONLY) = 5
fstat(5, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 8), ...}) = 0
openat(AT_FDCWD, "/dev/srandom", O_RDONLY) = -1 ENOENT (No such file or directory)
futex(0x7f338a020948, FUTEX_WAKE_PRIVATE, 2147483647) = 0
getrandom(

(BTW: this sequence of calls looks very much like it is coming from crypto/rand/rand_unix.c from openssl)

Sometimes it helps to simply press a couple of keys or move the mouse and the getrandom call continues. But other times it just seems to hang "forever".

My conclusion so far is that there is not enough entropy and the getrandom call blocks until there is.

So here are my questions:
1. Using curl at early boot to access an https link used to work just fine. Why not anymore? Were there any changes in the kernel, curl or openssl regarding getrandom?
2. What are my options to make curl work again at early boot?

Any help in any direction is welcome!

Cheers, Curly060 =;->

(*) I reboot very rarely, so I cannot tell exactly when this started to happen. I'd say this worked fine in June this year, so it must have happened somewhere between then and now.

Last edited by Curly060 (2018-11-18 00:58:00)

/dev/zero · 2018-10-23 06:08:01

Hi Curly060,

Curly060 wrote:

I have a custom initramfs hook which uses curl to make an HTTPS call.

Any chance you could provide a way to reproduce the problem? What's the hook, what are some minimum steps someone could take to generate the exact same error?

Curly060 wrote:

My conclusion so far is that there is not enough entropy and the getrandom call blocks until there is.

Yeah, that was my thought, too.

Curly060 wrote:

(*) I reboot very rarely, so I cannot tell exactly when this started to happen. I'd say this worked fine in June this year, so it must have happened somewhere between then and now.

Keep in mind with Arch Linux, it isn't happy to go long periods of time without upgrading. Arch Linux is happiest when it gets a full system upgrade at least every week, and reboot as soon as you see a new kernel rock up.

There are other distros out there in the world if you need something that will tolerate neglect

seth · 2018-10-23 07:14:59

https://www.archlinux.org/packages/extr … 4/haveged/

V1del · 2018-10-23 07:28:53

Similar use case: https://bbs.archlinux.org/viewtopic.php?id=236992 apparently haveged doesn't work correctly (or p00h started it incorrectly, I'm not that versed in custom initramfs stuff), but yes this has been an issue in one form or the other since 4.16

/dev/zero · 2018-10-23 07:35:39

The problem with haveged is there isn't a lot of proof around its claims of cryptographic suitability. It is mainly just a convenient workaround for when you need to work with VMs that don't have /dev/random passed through to them from their host.

/dev/zero · 2018-10-23 07:36:44

/dev/zero wrote:

The problem with haveged is there isn't a lot of proof around its claims of cryptographic suitability. It is mainly just a convenient workaround for when you need to work with VMs that don't have /dev/random passed through to them from their host.

And having said that, the cases when that is required aren't too common, for example, if you're creating VMs using virt-manager you can easily pass through /dev/random from host.

/dev/zero · 2018-10-23 07:46:47

Furthermore, if we all think more about OP's problem, haveged is really an antipattern here.

Think about it: even for a VM, let alone a real machine, whatever entropy has been collected during the previous runtime should be preserved between boots.

The only time entropy is a problem is when a machine is booted for the very first time; this is why entropy problems crop up so often for VMs, they are being booted for the first time. Passing /dev/random from host bootstraps a new VM, letting it have more entropy than a new machine would, if it were a brand new physical machine.

OP says this has happened after reboot. That is a bit weird; it shows their machine is not preserving whatever entropy it managed to gather in between its previous boot and the most recent shutdown.

Seriously, haveged is not the answer here.

OP needs to provide instructions for reproducing the behaviour.

V1del · 2018-10-23 08:12:33

Please edit your post instead of triple posting when no one has answered in between.

Also, you might want to read through the thread I linked and through the link I linked in that thread. There was/is a bug in the kernel and the fix of which lead to having a starved entropy seed, even across reboots/shutdowns. Haveged alleviates that starvation, it is a workaround but it's a better workaround than having blocking calls on something you don't want to have blocked.

Last edited by V1del (2018-10-23 08:14:53)

/dev/zero · 2018-10-23 08:22:27

V1del wrote:

Also, you might want to read through the thread I linked and through the link I linked in that thread.

Or, you might quote the relevant parts.

Since there were exactly five posts in that thread, it wouldn't be hard to scrounge up the compelling evidence that you would like to refer to.

Thanks for contributing two of the posts in that thread, by the way. Your ability to take stabs at the truth and provide workarounds must be very useful to newcomers.

However, I don't see anywhere in that thread anything that explains why haveged is necessary for the current problem.

V1del wrote:

There was/is a bug in the kernel and the fix of which lead to having a starved entropy seed, even across reboots/shutdowns. Haveged alleviates that starvation, it is a workaround but it's a better workaround than having blocking calls on something you don't want to have blocked.

Citation needed.

Last edited by /dev/zero (2018-10-23 08:41:16)

seth · 2018-10-23 12:34:46

The linked thread links down to https://bugs.archlinux.org/task/58355 and according to the last post there, https://git.kernel.org/pub/scm/linux/ke … b15f596ee0 might resolve this better, but I assume is in 4.19 only.

Also feel free to read https://bugzilla.redhat.com/show_bug.cgi?id=1572944#c7 - there're three interesting links in that post…

/dev/zero · 2018-10-23 19:39:30

seth wrote:

The linked thread links down to https://bugs.archlinux.org/task/58355 and according to the last post there, https://git.kernel.org/pub/scm/linux/ke … b15f596ee0 might resolve this better, but I assume is in 4.19 only.
Also feel free to read https://bugzilla.redhat.com/show_bug.cgi?id=1572944#c7 - there're three interesting links in that post…

Thanks, that is interesting to know.

However, it still isn't clear that this affects systems that have just rebooted, after already running for some time. Surely it is only for first-time boots? Furthermore, if it's a VM, and the user has control over how the VM is created (eg they are booting it on their own desktop, not using a cloud service provider), they can use virtio-rng to pass through /dev/random from the host.

There is a service systemd-random-seed which saves (only some of?) the current entropy to /var/lib/systemd/random-seed in between boots. I think it only saves 512 ~~bits~~ bytes, but that is more than enough to initialise /dev/urandom, which is what almost everything relies on, except when generating private keys, which surely does not need to be done every single time a machine boots.

Really, the links provided so far all just say what I've already said, unless I'm misunderstanding something. And that is possible. But the misunderstanding would have to be something like: the kernel is now preventing systemd-random-seed from doing its job; or simply no entropy is being generated soever. And I just don't see anything like those problems in these links.

So, again, haveged is an anti-pattern, Curly060 should tell us more about how to reproduce the problem. After all, no one else around here seems to be having this problem, so there is something Curly060 is not telling us.

Is it a VM or hardware? Is it first-time boot or were there really previous boots? What's in this custom initramfs hook? Does /var/lib/systemd/random-seed exist on the machine? What's the status of the systemd-random-seed service? How much entropy is being reported by /proc/sys/kernel/random/entropy_avail?

Rather than speculating about kernel bugs let's make sure we have our facts straight.

Last edited by /dev/zero (2018-10-23 20:01:24)

jjacky · 2018-10-23 20:24:18

/dev/zero wrote:

However, it still isn't clear that this affects systems that have just rebooted, after already running for some time. Surely it is only for first-time boots? Furthermore, if it's a VM, and the user has control over how the VM is created (eg they are booting it on their own desktop, not using a cloud service provider), they can use virtio-rng to pass through /dev/random from the host.

I believe using virtio-rng is indeed a solution for VMs, and that has been noted on some of the links posted above (don't remember which which one(s) exactly, sorry). The problem is really for either "real" systems or VM where for one reaosn or another such device isn't available...

/dev/zero wrote:

There is a service systemd-random-seed which saves (only some of?) the current entropy to /var/lib/systemd/random-seed in between boots. I think it only saves 512 bits, but that is more than enough to initialise /dev/urandom, which is what almost everything relies on, except when generating private keys, which surely does not need to be done every single time a machine boots.

(I believe it should save whatever size is returned by /proc/sys/kernel/random/poolsize -- or said size / 8, so usually 4096 bits or 512 bytes.)
You seem to forget we're talking about happenings inside the initramfs here, i.e. very early in the boot process, and before/aiming to mount the rootfs. Meaning said rootfs isn't available just yet, and that is where such seed would have been saved, so probably it can't be used just yet. (There might also be a case of VM where such seed doesn't exist, because the same image is used many times and then obviously you don't want to just use the same seed everywhere.)

More importantly though, what you're talking about is writing that random seed back to /dev/urandom, but as the man page states:

Writing to /dev/random or /dev/urandom will update the entropy pool with the data written, but this will not result in a higher entropy count. This means that it will impact the contents read from both files, but it will not make reads from /dev/random faster.

So even if that happens to be done in early boot, it wouldn't help wrt available entropy and blocking.
This is also why on some bug report someone is wondering if that service shouldn't use ioctl(RNDADDENTROPY) instead, to increment the entropy count. That should help, I think someone mentioned having set up such a system and it solved the issue/blocking. But obviously that might not be possible here early in the initramfs where that seed isn't yet available, as mentioned before.

Curly060 · 2018-10-23 21:46:18

Hi,

thanks for all the replies so far. Here are the necessary steps to reproduce the problem quite easily:

/etc/mkinitcpio.conf

MODULES=(usb_storage sd_mod)
BINARIES=(strace curl)
HOOKS=(base udev block autodetect modconf keyboard keymap my-custom-hook filesystems fsck)

Or in words: add curl and strace as binaries and add the my-custom-hook some point after modconf.

/etc/initcpio/install/my-custom-hook

#!/bin/bash

build() {
  add_checked_modules '/drivers/net/'
  add_binary "/usr/lib/initcpio/ipconfig" "/bin/ipconfig"

  # necessary to get DNS resolution
  add_file "/usr/lib/libnss_dns.so.2" "/usr/lib/libnss_dns.so.2" "$(stat -Lc %a /usr/lib/libnss_dns.so.2)"
  add_file "/usr/lib/libnss_files.so.2" "/usr/lib/libnss_files.so.2" "$(stat -Lc %a /usr/lib/libnss_files.so.2)"

  add_runscript
}

help() {
  cat <<HELPEOF
My test hook
HELPEOF
}

# vim: set ft=sh:

/etc/initcpio/hooks/my-custom-hook

#!/usr/bin/ash

run_hook() {
  ### Set up network
  # These variables will be parsed from /tmp/net-*.conf generated by ipconfig
  local DEVICE
  local IPV4ADDR IPV4BROADCAST IPV4NETMASK IPV4GATEWAY IPV4DNS0 IPV4DNS1
  local HOSTNAME DNSDOMAIN NISDOMAIN ROOTSERVER ROOTPATH
  local filename

  ipconfig "ip=dhcp"

  for conf in /tmp/net-*.conf; do
    [ -f "$conf" ] && . "$conf"
  done
  echo "nameserver $IPV4DNS0" > /etc/resolv.conf

  echo "about to curl..."
  strace /usr/bin/curl -k -v --connect-timeout 9 -m 5 https://www.google.de/
  return 0
}

Don't forget to

mkinitcpio -p linux

kernel command line
You need to add ip=dhcp to the kernel command line.

A couple of words to the discussion:

This happens with cold boots and reboots
This is a physical machine, not a VM
rootfs is not yet available. In fact, the whole point of that curl command is to get a key to unlock the rootfs (spare me the details, please )
Wouldn't https://git.kernel.org/pub/scm/linux/ke … 9da94bdd0b be exactly the right thing to get high enough entropy at least for a reboot? But I am not sure if this is active or how to activate it
Yes, I know I need to update Arch Linux frequently and I usually do :-)

Cheers, Curly060 =;->

Last edited by Curly060 (2018-10-23 21:47:21)

Scimmia · 2018-10-24 00:20:58

/dev/zero wrote:

seth wrote:
The linked thread links down to https://bugs.archlinux.org/task/58355 and according to the last post there, https://git.kernel.org/pub/scm/linux/ke … b15f596ee0 might resolve this better, but I assume is in 4.19 only.
Also feel free to read https://bugzilla.redhat.com/show_bug.cgi?id=1572944#c7 - there're three interesting links in that post…
Thanks, that is interesting to know.
However, it still isn't clear that this affects systems that have just rebooted, after already running for some time. Surely it is only for first-time boots?

Nope. Read again.

V1del · 2018-10-24 00:30:28

Also before you test this, as pointed out in the thread (and the CVE's and the bug reports...) this can be hardware specific and it doesn't have to affect your particular setup.

We know the issue exists, we know that rebooting doesn't have an effect (or that the system unit you point out has any positive effect on it, from what I'm seeing that's enabled by default and it still happens with non initramfs systems) and we know the symptoms point to this exact issue with all likelihood and we know haveged is a stop gap fix (if you don't trust haveged, the other option would be rng-tools and enabling CPU random seed (or the mentioned kernel parameter with future kernels) however if you aren't trusting haveged you have no reason to trust this either).

Last edited by V1del (2018-10-24 01:15:28)

Curly060 · 2018-11-18 00:57:22

With the new 4.19.x kernel a

random.trust_cpu=on

on the kernel command line fixes the issue for me!

/dev/zero · 2018-11-18 21:22:09

Curly060 wrote:

With the new 4.19.x kernel a
random.trust_cpu=on
on the kernel command line fixes the issue for me!

Thanks, this is interesting to know.

It still surprises me how this could be a problem. However, after participating in this thread, I started paying more attention to how much entropy is actually available on my workstations, using a meter in my dwm status bar. Indeed, entropy does not seem to be preserved between reboots, or at least not to the extent I would have anticipated.

It's good that we can now choose particular behaviours depending on factors like whether we actually do trust the cpu, and security vs economy trade-offs.

loqs · 2018-11-18 23:29:28

No entropy is preserved between boots. Systemd saves a random seed on shutdown which is written back in early boot but that write back is not credited with any entropy see https://github.com/systemd/systemd/pull/10621

/dev/zero · 2018-11-18 23:43:29

loqs wrote:

No entropy is preserved between boots. Systemd saves a random seed on shutdown which is written back in early boot but that write back is not credited with any entropy see https://github.com/systemd/systemd/pull/10621

Huh. I could have sworn I read somewhere that you only need to worry about entropy on the first boot because whatever gets generated during runtime will be written to disk during shutdown for re-use on the next boot. I must have misunderstood O_o

Arch Linux

#1 2018-10-22 20:34:35

[SOLVED] curl in initramfs hangs at getrandom

#2 2018-10-23 06:08:01

Re: [SOLVED] curl in initramfs hangs at getrandom

#3 2018-10-23 07:14:59

Re: [SOLVED] curl in initramfs hangs at getrandom

#4 2018-10-23 07:28:53

Re: [SOLVED] curl in initramfs hangs at getrandom

#5 2018-10-23 07:35:39

Re: [SOLVED] curl in initramfs hangs at getrandom

#6 2018-10-23 07:36:44

Re: [SOLVED] curl in initramfs hangs at getrandom

#7 2018-10-23 07:46:47

Re: [SOLVED] curl in initramfs hangs at getrandom

#8 2018-10-23 08:12:33

Re: [SOLVED] curl in initramfs hangs at getrandom

#9 2018-10-23 08:22:27

Re: [SOLVED] curl in initramfs hangs at getrandom

#10 2018-10-23 12:34:46

Re: [SOLVED] curl in initramfs hangs at getrandom

#11 2018-10-23 19:39:30

Re: [SOLVED] curl in initramfs hangs at getrandom

#12 2018-10-23 20:24:18

Re: [SOLVED] curl in initramfs hangs at getrandom

#13 2018-10-23 21:46:18

Re: [SOLVED] curl in initramfs hangs at getrandom

#14 2018-10-24 00:20:58

Re: [SOLVED] curl in initramfs hangs at getrandom

#15 2018-10-24 00:30:28

Re: [SOLVED] curl in initramfs hangs at getrandom

#16 2018-11-18 00:57:22

Re: [SOLVED] curl in initramfs hangs at getrandom

#17 2018-11-18 21:22:09

Re: [SOLVED] curl in initramfs hangs at getrandom

#18 2018-11-18 23:29:28

Re: [SOLVED] curl in initramfs hangs at getrandom

#19 2018-11-18 23:43:29

Re: [SOLVED] curl in initramfs hangs at getrandom

Board footer