[Solved] pacman and unmounted /boot

Jens Clasen · 2018-02-18 08:50:15

Good morning forum!

Today I had my daily wtf a little early:

Being a bit ancient (and new to arch), I normally have /boot unmounted. It's listed in fstab but only like this:

/dev/sda1		/boot		ext3		noauto,noatime	0 2

Now four things came together:

pacman -Syu
a kernel update
me without my first cup of coffee, saying yes to the install without thinking
the daily wtf, being pacman triggering a warning instead of an error, because of the external /boot partition, resulting in a broken system.

Afterwards I had a kernel image inside the wrong /boot folder (or to be precise: in the right folder, but in the wrong partition) and I could not mount the right partition (ext3 was an unknown file system). Even worse: A reboot wouldn't have worked, since the kernel modules for my old kernel on /dev/sda1 are gone.

I had to boot from the live CD to fix that. (Already done, so there's no current problem.)

What I don't understand is:

1) Why the hell is an unavailable target directory only a warning and not an error? Especially since pacman (or some kind of install hook) could easily solve these kind of problems itself by just running "mount /boot".

2) How can those problems be avoided?

Having /boot always mounted readonly seems to be one way, since the update would probably fail with a real error in this case. I'm just not sure, if this wouldn't leave me in a worse state, with a half updated system.

Wrapping pacman -Syu in a script, which would always mount /boot writable before installing and unmount afterwards could be an idea. It just seems to be a bit of an overkill to do so. And: there's the risk, that I'll forget to even use the script instead of pacman -Syu.

Are there any kind of system specific hooks which I could use to tell pacman to do that before installing anything?

Any ideas or suggestions?

Regards, Jens

Last edited by Jens Clasen (2018-03-04 23:04:47)

lo1 · 2018-02-18 09:21:44

Have a look at /usr/share/libalpm/hooks (probably you would want to edit your 90-linux.hook). The easiest way I can think of is just edit it like

[Action]
Description = Updating linux initcpios...
When = PostTransaction
Exec = mount /your/boot/partition && /usr/bin/mkinitcpio -p linux

but I frankly don't know how safe would it be. Read alpm-hooks if you were thinking of a most consistent solution.

Also note that your hooks may be overriden so put them in /etc when you're sure they work good (thanks to progandy).

Last edited by lo1 (2018-02-18 09:21:56)

WorMzy · 2018-02-18 10:19:01

By "unavailable target directory" warning message, I suspect you are referring to this message: "WARNING: /boot appears to be a separate partition but is not mounted.", but if not, please clarify on this.

This message is not printed by pacman (at least, not directly). Pacman doesn't care whether your partitions are mounted or not. If you tell it to install the linux package to the wrong partition, it will go ahead and do that. The message is actually included as part of the linux package: https://git.archlinux.org/svntogit/pack … ages/linux and is presumably only included so that you can check and fix the problem before you reboot (downgrading to the old kernel so you can mount your boot partition if necessary).

If you insist on keeping boot unmounted for whatever reason, then, rather than editing or creating pacman hooks (which wouldn't help with the kernel images being installed to the wrong place), you could create an alias/function/wrapper script which mounts your boot partition before upgrading, then unmounts it after the fact.

Head_on_a_Stick · 2018-02-18 10:21:16

Jens Clasen wrote:

How can those problems be avoided?

Use a GPT disk & systemd-boot with no fstab at all — systemd will unmount /boot automatically after 120 seconds and will mount it on-demand for kernel upgrades and suchlike automagically.

Is your machine UEFI capable?

It does make boot management significantly easier, IMO.

seth · 2018-02-18 11:29:48

Is that actually GPT magic?
You could achieve this with the x-systemd.automount and x-systemd.idle-timeout parameters in the fastab entry for your boot partition. The only thing GPT/UEFI related here is afaiu that systemd knows for pretty sure what the boot partition is...

Trilby · 2018-02-18 12:15:01

seth wrote:

You could achieve this with the x-systemd.automount and x-systemd.idle-timeout parameters

I'm not sure I fully understand the reasoning for not having the boot partition mounted, but whatever it is, I'm pretty sure these options would not meet the criteria. The only reasons I could even imagine for not having boot mounted would be to prevent undesired writes/corruption of any data on it. If systemd will happily mount it behind the scenes anytime anything tries to write to it, this would defeat the purpose of having it unmounted.

seth · 2018-02-18 12:33:07

I rather wondered about the systemd/GPT assertion, but yes: anything that automounts a deliberately unmounted partition would rather contradict that deliberation.
Where to tackle the condition is a matter of the reason to not have /boot mounted itfp. - likely mkinitcpio, maybe pacman.

Head_on_a_Stick · 2018-02-18 13:03:03

seth wrote:

Is that actually GPT magic?

I think so, yes:

https://www.freedesktop.org/software/sy … rator.html

seth wrote:

You could achieve this with the x-systemd.automount and x-systemd.idle-timeout parameters in the fastab entry for your boot partition

Granted, yes, but with a GPT disk the partition codes can be used to define the ESP (and hence /boot) along with /home, any swap and the root partition and allow the machine to be booted normally even in the absence of /etc/fstab

Trilby wrote:

If systemd will happily mount it behind the scenes anytime anything tries to write to it, this would defeat the purpose of having it unmounted.

That being the case, why have the developers added it as a function?

Allan · 2018-02-18 13:25:21

Creating a pre-transaction hook that aborts pacman updates if both /boot is unmounted and you need access to boot is fairly easy...

eschwartz · 2018-02-18 14:05:47

This would be so easy to solve, by using a bootloader that does not require you to do silly things like mounting /boot as a separate partition...

grub and refind both support loading a kernel from partitions other than the ESP. It would also be nice if the UEFI spec didn't standardize on the worst filesystem ever...

Or even, don't unmount /boot without a compelling reason.

Jens Clasen wrote:

1) Why the hell is an unavailable target directory only a warning and not an error? Especially since pacman (or some kind of install hook) could easily solve these kind of problems itself by just running "mount /boot".

pacman itself does not care and cannot care. This is basic filesystems 101, an unmounted partition is not an error and the filesystem will just write to a directory instead. Maybe you want to complain to the people who made filesystems that they are doing it wrong, because clearly what *everyone* wants is that the partition they explicitly de-mounted and told never to mount itself, should do so anyway.

pacman, or Arch, does however take an explicit stance on people who "saying yes to the install without thinking" and then don't read warnings and other output. Quite simply, we don't support that.

So why should pacman take away choice from the user in a situation where they've gone out of their way to add some highly irregular configuration options? As Allan said, it is trivial to add a PreTransaction hook to correspond to your fstab entry.

Trilby · 2018-02-18 14:20:30

Head_on_a_Stick wrote:

That being the case, why have the developers added it as a function?

You want me to speculate on why someone else implemented something I'm critical of? That simply cannot lead to productive discussion. You'd have to ask those developers why they did this.

Last edited by Trilby (2018-02-18 14:20:47)

Jens Clasen · 2018-02-18 16:28:32

First of all, a heartfelt "thank you" to all of you, for taking the time to leave a comment.

The system in question isn't UEFI capable, therefor the GPT approach is probably out of question. Thanks for the tip anyway. I didn't knew of systemd's automount feature and I learned something new from reading about it.

As for the question, why I would have /boot unmounted or even on a separate partition, you're partly right with your assumption. When I started out using Linux in 1995, systems with a lot of separate partitions where more or less the norm and keeping /boot unmounted was common practice - exactly for the mentioned reason of no unwanted damage to any files there. I simply kept that because it worked. There's no reason to fix something, which isn't broken - even if it isn't common practice any longer.

I reduced the number of partitions in use over the years, but that came back to bite me in the ass just a couple of weeks ago, when I wanted to switch between Linux distributions. My quintessence was, that the next linux setup I'm doing will have at least separate partitions for the user files, for web and for database data again. But that's just a side note.

Thinking a bit further about using the hook to automatically mount /boot on install of the linux package, I feel that this approach probably isn't the right solution, since it would contradict the reason for keeping /boot unmounted. Let's be honest: at least 50% of the mistake this morning was saying yes to the install at all, only the rest was the problem, that the install didn't fail.

What I still fail to see is, why

WARNING: /boot appears to be a separate partition but is not mounted.

is a warning and not an error, though.

I mean, that sounds like "I know, I will break your system now, but I'll do so anyway. *stick tongue out*" and is not really nice - regardless, where the warning comes from. (I stand corrected, that it's not pacman, but the package, though.)

As soon as there is a different FS in use for /boot, than those already mounted, the kernel will not have the necessary modules loaded to even fix that without a live CD. And while you can downgrade a package in arch, you'll need to find and manually download the necessary package. There's no real support for a downgrade in pacman's repository, therefor that feels like a discouraged approach.

Given all that, I'm coming back to finding a solution to prevent pacman from even doing the update, if the condition "/boot needs to be available as configured" is not met.

What do you think about an alpm-hook like that:

[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Operation = Remove
Target = boot/*
Target = usr/lib/initcpio/*

[Action]
Description = Check if /boot is available and abort otherwise
When = PreTransaction
Exec = mount | grep /boot >/dev/null
AbortOnFail

The whole thing is untested, but as I'm reading it, it should prevent unwise installs without having /boot available. What I still do not really understand is, in what state I'd be afterwards. Would the whole system update fail or would the update simply run against a wall, when installing the linux package?

@WorkMzy: You mentioned, a hook wouldn't help with the kernel images being installed to the wrong place. I'm not sure, I understand. Is there any other place, I have to think about, beside /boot?

Regards, Jens

// EDIT: Reading the thread again, I see that I missed Allen's suggestion of doing exactly the same thing as my pre transaction trigger above. Thanks for your suggestion, anyhow!

// EDIT 2: I'd like to comment on a part of your answer, Eli:

Eschwartz wrote:

pacman, or Arch, does however take an explicit stance on people who "saying yes to the install without thinking" and then don't read warnings and other output. Quite simply, we don't support that.

Nobody is expecting arch to do so - least of all I. But: I saw that warning. At that point it was already more or less too late. And at that point a warning shouldn't be a Warning, IMHO. On the other hand: If what you're saying is true, maybe one should remove the warning? I mean, it contradicts the "not supporting people saying yes without thinking" policy.

Last edited by Jens Clasen (2018-02-18 16:53:53)

Trilby · 2018-02-18 17:14:28

Jens Clasen wrote:

What I still fail to see is, why [warning] is a warning and not an error, though.

Because "Unix was not designed to stop you from doing stupid things ..."

Although WorMzy mentioned it, the point seems to have been passed by quickly: the proper fix if you do find yourself in this situation again is much simpler than what you ended up doing:

pacman -U /var/cache/pacman/pkg/linux-<whatever>.pkg.tar.gz

Then you've basically done the equivalent of skipping the kernel update and you are in a position to chose not to do it at that time, or to mount your root partition and complete the update.

eschwartz · 2018-02-18 17:20:37

Jens Clasen wrote:

First of all, a heartfelt "thank you" to all of you, for taking the time to leave a comment.
The system in question isn't UEFI capable, therefor the GPT approach is probably out of question. Thanks for the tip anyway. I didn't knew of systemd's automount feature and I learned something new from reading about it.

Then the one understandable reason for having a separate boot partition, that you're using some badly designed minimal UEFI bootloader that doesn't comprehend modern filesystems, is apparently not even the cause.

As for the question, why I would have /boot unmounted or even on a separate partition, you're partly right with your assumption. When I started out using Linux in 1995, systems with a lot of separate partitions where more or less the norm and keeping /boot unmounted was common practice - exactly for the mentioned reason of no unwanted damage to any files there. I simply kept that because it worked. There's no reason to fix something, which isn't broken - even if it isn't common practice any longer.

That logic leads to having a separate partition for every directory. The usual advice there, was separate /var to prevent ballooning logfiles eating 200GB of your HDD and leading to out of space errors.

I reduced the number of partitions in use over the years, but that came back to bite me in the ass just a couple of weeks ago, when I wanted to switch between Linux distributions. My quintessence was, that the next linux setup I'm doing will have at least separate partitions for the user files, for web and for database data again. But that's just a side note.

/boot is not user data, and it is not transferable between distros. So there is no reason to have a shared /boot for different distros. Ever. If you're using some-UEFI-bootloader-I-hate, with a shared ESP, each distro should live in a subdirectory of the ESP.

What I still fail to see is, why
WARNING: /boot appears to be a separate partition but is not mounted.
is a warning and not an error, though.
I mean, that sounds like "I know, I will break your system now, but I'll do so anyway. *stick tongue out*" and is not really nice - regardless, where the warning comes from. (I stand corrected, that it's not pacman, but the package, though.)
As soon as there is a different FS in use for /boot, than those already mounted, the kernel will not have the necessary modules loaded to even fix that without a live CD. And while you can downgrade a package in arch, you'll need to find and manually download the necessary package. There's no real support for a downgrade in pacman's repository, therefor that feels like a discouraged approach.

As the warning is provided by an install script, not a pacman hook, it cannot abort the transaction.

And, there is no reason to abort anyway. You don't need to downgrade, just manually mount /boot and reinstall the *new* kernel package.

So, why should it include a hook instead, to abort the transaction? That would completely block system updates, and most people don't even reboot when the kernel is updated. (Note that this is technically just as broken, see FS#16702.)

Given all that, I'm coming back to finding a solution to prevent pacman from even doing the update, if the condition "/boot needs to be available as configured" is not met.
What do you think about an alpm-hook like that:
[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Operation = Remove
Target = boot/*
Target = usr/lib/initcpio/*

[Action]
Description = Check if /boot is available and abort otherwise
When = PreTransaction
Exec = mount | grep /boot >/dev/null
AbortOnFail
The whole thing is untested, but as I'm reading it, it should prevent unwise installs without having /boot available. What I still do not really understand is, in what state I'd be afterwards. Would the whole system update fail or would the update simply run against a wall, when installing the linux package?

Ew, use findmnt /boot instead (just like the linux install script uses!) and avoid both the deprecated use of `mount`, and a useless use of grep. Also note that you need to use Exec = /bin/sh -c 'commands to run' if you want shell syntax, as Exec uses neither a shell nor PATH lookup (but /bin/sh -c provides both).

I guess you want shell syntax in order to quiet findmnt when it successfully finds the mountpoint. I cannot think of a way to totally quiet findmnt and only care about the retun code.

@WorkMzy: You mentioned, a hook wouldn't help with the kernel images being installed to the wrong place. I'm not sure, I understand. Is there any other place, I have to think about, beside /boot?
Regards, Jens
// EDIT: Reading the thread again, I see that I missed Allen's suggestion of doing exactly the same thing as my pre transaction trigger above. Thanks for your suggestion, anyhow!
// EDIT 2: I'd like to comment on a part of your answer, Eli:
Eschwartz wrote:
pacman, or Arch, does however take an explicit stance on people who "saying yes to the install without thinking" and then don't read warnings and other output. Quite simply, we don't support that.
Nobody is expecting arch to do so - least of all I. But: I saw that warning. At that point it was already more or less too late. And at that point a warning shouldn't be a Warning, IMHO. On the other hand: If what you're saying is true, maybe one should remove the warning? I mean, it contradicts the "not supporting people saying yes without thinking" policy.

But they did think. They chose to say yes, with the knowledge that they weren't *positive* what the results would be but that they were going to take a look at pacman's output to see if there is any insight they missed.

It is *not* too late, to reinstall linux with `sudo mount /boot && sudo pacman -S linux`

Jens Clasen · 2018-03-04 23:04:25

First of all, sorry for being so late in coming back to this matter. Real-live took over once again and I didn't spent much time with tinkering with linux.

Thank you for your comments here. Your comments teached me, that I needed to read a bit more, before trying to take part in a discussion about the way pacman works. I read the bug report about the kernel upgrade procedure. Exactly the problem described there was hitting me as well. "mount /boot" was impossible due to missing modules for the file system.

For now I solved the problem using two different approaches together:

1) I installed kernel modules hook from AUR.

2) I created a new hook based on the examples for libaplm:

[Trigger]
Operation = Install
Operation = Upgrade
Operation = Remove
Type = File
Target = boot/*
Target = usr/lib/initcpio/*

[Action]
When = PreTransaction
Exec = /bin/sh -c 'findmnt /boot &>/dev/null || ( printf "error: /boot not mounted\n"; exit 1 )'
AbortOnFail

I did the second update today running this config and currently it seems, as if the approach works for me.

I'm not perfectly happy with those AUR hooks, since I'd rather prefer to either mark a given kernel version or automatically mark the currently running as stable/unremovable and keep it around for fallback purposes, but I did not find a good solution for this yet, beside doing this manually. For now, I guess I can live with what I have.

As for the FS layout part of the discussion:

For a number of years, /boot was a single partition, /home, /var, /tmp and / where others. Later that came down to having /boot and /. At least for me and that's what I'm running with, at the moment. I did see /var getting too large and I do remember a server, where /tmp managed fill up the whole system, though. I remember another server, where a certain user managed to do the same with a download in /home. Therefor, the separate partition approach for a couple of things has a certain benefit in my book.

If I should have to decide that again, I'd probably at least keep /home and the databases in /var separate. I do understand your arguments regarding /boot, but my conclusion is a bit different. The question coming to my mind is, if some kind of readonly mounting for /boot and /usr wouldn't have some benefits. Having mixed access to both was the root cause of my WTF mentioned above...

What I still do not understand, though: Why should an install script not be able to fail a transaction, when it detects some kind of missing prerequisite? Isn't that some kind of flaw in the concept of those things?

Regards, Jens

Last edited by Jens Clasen (2018-03-04 23:28:41)

Arch Linux

#1 2018-02-18 08:50:15

[Solved] pacman and unmounted /boot

#2 2018-02-18 09:21:44

Re: [Solved] pacman and unmounted /boot

#3 2018-02-18 10:19:01

Re: [Solved] pacman and unmounted /boot

#4 2018-02-18 10:21:16

Re: [Solved] pacman and unmounted /boot

#5 2018-02-18 11:29:48

Re: [Solved] pacman and unmounted /boot

#6 2018-02-18 12:15:01

Re: [Solved] pacman and unmounted /boot

#7 2018-02-18 12:33:07

Re: [Solved] pacman and unmounted /boot

#8 2018-02-18 13:03:03

Re: [Solved] pacman and unmounted /boot

#9 2018-02-18 13:25:21

Re: [Solved] pacman and unmounted /boot

#10 2018-02-18 14:05:47

Re: [Solved] pacman and unmounted /boot

#11 2018-02-18 14:20:30

Re: [Solved] pacman and unmounted /boot

#12 2018-02-18 16:28:32

Re: [Solved] pacman and unmounted /boot

#13 2018-02-18 17:14:28

Re: [Solved] pacman and unmounted /boot

#14 2018-02-18 17:20:37

Re: [Solved] pacman and unmounted /boot

#15 2018-03-04 23:04:25

Re: [Solved] pacman and unmounted /boot

Board footer