[solved] updating kernel makes "incomplete" initramfs

jjacky · 2012-02-29 21:12:47

Hi,

Okay so this happened to me last time already[1], but I wasn't sure I hadn't done something. But it happened again: when I upgraded the kernel, I did get an error message during the mkinitcpio run:

[2012-02-29 21:17]   -> Parsing hook: [autodetect]
[2012-02-29 21:17] xargs: modprobe: No such file or directory

Other than that nothing, that is no warnings/errors reported at any point. This resulted in an "incomplete" initramfs, where at least modules needed for my HDDs where missing[2], hence no way to boot (well, no way to mount root).

Luckily the fallback doesn't have this problem, and all I needed to do to fix it was to run mkinitcpio again. That is, this is all I did:

# mkinitcpio -p linux

And then it ran, without any errors. (And the newly created initramfs had all the required modules.)

I have no idea why there is a problem when it is run upon upgrading, or what I need to do to fix this. The only difference I can think of between the two runs is that, obviously, the first one occured while the old kernel was still running, whereas the one I did manually was with the new kernel running, but that seems pretty normal/expected. (Also, I'm not sure when that error occurs, as AFAICS there's no "xargs" in the autodetect hook it seems... ?)

Any ideas of what's going on and/or how to fix it?
Thanks,
-jacky

[1] looking at pacman.log, it seems to have happened 3 times actually: 3.2.5-1 -> 3.2.6-2; 3.2.6-2 -> 3.2.7-1; and today's 3.2.7-1 -> 3.2.8-1
I only remember going through this once before, but I trust pacman's log more than my memory.

[2] I actually still have the "incomplete" initramfs this time, so in case it matters here's the full list of missing files:

./lib/modules/3.2.8-1-ARCH/kernel/drivers/usb/host/ehci-hcd.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/usb/host/uhci-hcd.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/cdrom/cdrom.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/ata/ahci.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/ata/libahci.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/ata/pata_marvell.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/ata/pata_acpi.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/ata/libata.ko.gz
./lib/modules/3.2.8-1-ARCH/kernel/drivers/scsi/sr_mod.ko.gz

Last edited by jjacky (2012-03-11 16:51:29)

karol · 2012-02-29 21:28:06

Did you upgrade just the kernel or the whole system?

jjacky · 2012-02-29 21:32:39

The whole system, each time.

Edit: Yeah I said updating the kernel as that's obviously the "trigger" but I always did full system upgrade.

Last edited by jjacky (2012-02-29 21:34:25)

falconindy · 2012-02-29 23:22:23

Figure out why modprobe seems to be missing.... run 'ls -l $(type -ap modprobe)' from a bash shell and post the output.

jjacky · 2012-02-29 23:46:12

There you go:

$ ls -l $(type -ap modprobe)
lrwxrwxrwx 1 root root 15 Feb 16 01:03 /sbin/modprobe -> ../usr/bin/kmod
$ modprobe -V
kmod version 5
$ pacman -Qs kmod
local/kmod 5-4
    Linux kernel module handling

jjacky · 2012-03-01 17:59:18

I see linux-3.2.9 is in testing, so it probably won't be long until it hits core. Any ideas of what I should try then, as I'm fairly confident this will happen again? I'll run mkinitcpio manually before rebooting of course, but anything I could/should try before that, and/or after in case it fails ?

jjacky · 2012-03-09 12:55:31

Alright, so after some more investigations here's what I found out:

- I don't do my system upgrades using pacman, but a little app I made, kalu. As far as this goes, it pretty much does exactly what a pacman -Syu does. However, I found out that using pacman things worked, but not with kalu.

- kalu does not actually do the upgrade, another binary (kalu-dbus) does (so what runs as root contains only the minimum required). However, if I start it manually (i.e. I run sudo kalu-dbus from a terminal before using kalu), then it works fine as well! This confirmed what I suspected might be the only difference between pacman & kalu-dbus: the later is run automaticaly by dbus, not manually from a terminal. (Although I don't know why it would matter.)

The problem

After some more testing, here's what I found, though I cannot explain it (I'd appreciate any insight if you have some). So I created an empty package, with the following post_install hook:

echo PATH=$PATH
/tmp/gp

/tmp/gp is a binary whose code is this:

#include <stdlib.h>
#include <stdio.h>

int main()
{
 const char *path = getenv("PATH");
 printf ("PATH=%s\n", path);
 return 0;
}

When installing the package using pacman, or via a manually launched kalu-dbus, I get the same PATH listed twice. However, when kalu-dbus has been started via dbus, I get the following:

PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
PATH=(null)

I believe this is also why I get the "modprobe not found" error. (I did first run `which modprobe` on the post_install hook, and got an error "no modprobe in (null)" which led to this last test)

Now, I cannot explain why `echo $PATH` does have a path, while getenv() fails to find one. If there is something I need to do e.g. in the .service file for dbus, I'd be happy to, but I couldn't find anything.

A "fix" in mkinitcpio?

What I know - although that might not be the best of solutions - is that if e.g. mkinitcpio would run `declare -x PATH=$PATH` first, things would work. And I see that there's already a PATH thing in there:

# Add /{,usr}/sbin to path
# works around undetected problems like in #8448
PATH=$PATH:/sbin:/usr/sbin

Now, while I'm not sure this could be called a bug of mkinitcpio per-se, could it be possible to change that line from "PATH=..." to "declare -x PATH=..." ?
I don't think it would cause problems, but it would fix this issue I've been experiencing

Thanks,
-jjacky

PS: I decided to post here & not open a bug report, since it's related and because I'm not sure it really can be called a bug of mkintcpio. If I should open a bug report instead/as well, let me know.

Lone_Wolf · 2012-03-10 12:02:33

Possibly kalu-dbus when started automatically runs in a separate session where the environment variables are not set correctly.

What is the exact command dbus uses to start kalu-dbus ?

jjacky · 2012-03-10 12:47:16

Lone_Wolf wrote:

Possibly kalu-dbus when started automatically runs in a separate session where the environment variables are not set correctly.
What is the exact command dbus uses to start kalu-dbus ?

The .service file is as follows:

[D-BUS Service]
Name=org.jjk.kalu
Exec=/usr/bin/kalu-dbus
User=root

Looking some more into this, it seems that (for security purposes) dbus might clear all environment variables when it starts something, which would explain it. (I still don't know why a "echo $PATH" from a bash script works/how it gets the path, though).
Which means that, even though I didn't try it, if one was to setup somethig to be started via dbus and then run `pacman -Syu` the same problem/error would occur, for the same reason. Or that any libalpm frontend that isn't CLI, and doesn't want to have the main process run with root privileges and therefore uses DBus to start another binary to do the libalpm interaction (which I believe is the right way to handle such a thing), would face the same issue.

falconindy · 2012-03-10 13:50:11

jjacky wrote:

A "fix" in mkinitcpio?
What I know - although that might not be the best of solutions - is that if e.g. mkinitcpio would run `declare -x PATH=$PATH` first, things would work. And I see that there's already a PATH thing in there:
# Add /{,usr}/sbin to path
# works around undetected problems like in #8448
PATH=$PATH:/sbin:/usr/sbin
Now, while I'm not sure this could be called a bug of mkinitcpio per-se, could it be possible to change that line from "PATH=..." to "declare -x PATH=..." ?
I don't think it would cause problems, but it would fix this issue I've been experiencing
Thanks,
-jjacky
PS: I decided to post here & not open a bug report, since it's related and because I'm not sure it really can be called a bug of mkintcpio. If I should open a bug report instead/as well, let me know.

There's only so much broken user behavior I'm willing to work around. Environment vars only need to be exported once -- if you change them, they still retain the 'exported' attribute, and PATH gains this attribute in /etc/profile. If your PATH isn't marked for export, fix that yourself.

edit: I also have zero interest in supporting any kind automated upgrade toy.

Last edited by falconindy (2012-03-10 13:51:15)

jjacky · 2012-03-10 14:09:45

falconindy wrote:

There's only so much broken user behavior I'm willing to work around. Environment vars only need to be exported once -- if you change them, they still retain the 'exported' attribute, and PATH gains this attribute in /etc/profile. If your PATH isn't marked for export, fix that yourself.

But my PATH *is* marked for export (in /etc/profile). As I said, the thing is that when dbus starts something it automatically unsets all environment variables, so I don't think this falls under broken user behavior.

falconindy wrote:

edit: I also have zero interest in supporting any kind automated upgrade toy.

Right, except that as I said in my previous post, I think this is something that would happen for anything launched via dbus, which seems to me to be the correct way for any GUI frontend to safely use libalpm, no?

falconindy · 2012-03-10 14:36:40

How did you come to that conclusion? ALPM is a C library, and has zero affiliation with DBus (nor does DBus make sense for something that isn't a long running process). The correct way for any frontend to use alpm is to use the C library.

jjacky · 2012-03-10 15:08:24

Yes, but if the frontend is one with a GUI, isn't it better that it doesn't run with root priviliges, so all the GUI stuff and linkage to GTK/QT/etc is all running under the user account, and have a second binary that will run as root and deal with libalpm, with only minimum required code (and no GUI). In which case, using dbus & polkit seems the right way to handle this (start that second binary w/ root privileges and check auth), no?

This is what's happening here: kalu is the program with the GUI and stuff, and it starts (through dbus) kalu-dbus, which runs as root & uses libalpm to do the sysupgrade.

As for the issue, I think another way I could solve this is to have dbus not start kalu-dbus but a bash script that would do 2 things: source /etc/profile, and start kalu-dbus. Sourcing /etc/profile will restore the path, and should solve the issue. I guess it might be better, since it would make the PATH available for everything, not just mkinitcpio.

The only thing I still cannot understand in this, remains why (from "dbus-started environment") bash scripts (e.g. mkinitcpio) have their $PATH variable set, while C program (e.g. which) do not have a PATH set in their environement. Any idea about this?

falconindy · 2012-03-10 15:17:07

Sounds to me like you're trying to reimplement packagekit. Go see what they do about keeping a sane environment.

jjacky · 2012-03-10 15:49:05

Well I'm not trying to do a full frontend at all, it only does a sysupgrade. Anyhow, thanks, I looked at packagekit, and apparently they do use dbus/polkit as well, and have the same issue. What they do is simply set the path quite strictly, which I'm not sure is a good/better idea:

PK_BACKEND_DEFAULT_PATH = "/bin:/usr/bin:/sbin:/usr/sbin"

/* PATH might have been nuked by D-Bus */
g_setenv ("PATH", PK_BACKEND_DEFAULT_PATH, FALSE);

So I think I'll just use a bash script to source /etc/profile, I feel it's better than forcing a path.

Arch Linux

#1 2012-02-29 21:12:47