Autoremoval of old kernel-stuff on kernel update

Lorddusty · 2014-10-24 11:00:05

Hi,

when I update the kernel-packages, the older version of the kernel is automatically removed, including all modules etc. This can (and does in my case) lead to some weired issues. Is there a way to stop removing the older version of a package when a newer version is installed, so that I can manually remove older kernel-packages as soon as the machine is bootet into the new kernel? If not, it might be a good idea to implement something like that.

Any hint would be appreciated.

Thank you and kind regards
Jesn

drcouzelis · 2014-10-24 12:12:27

It's the package manager's job to remove the old version of the package and install the new version (which it does). Updating the kernel requires a reboot (of course). So, simply don't update the kernel until you are ready to reboot!

~~What weird issues are you experiencing?~~ (It's all mentioned in the bug report that WorMzy posted.)

Last edited by drcouzelis (2014-10-24 12:37:26)

WorMzy · 2014-10-24 12:18:57

Lorddusty wrote:

it might be a good idea to implement something like that.

One day, perhaps...

Last edited by WorMzy (2014-10-24 12:19:56)

Lorddusty · 2014-10-24 19:36:14

Hi,

the weired issue I stumbled over this morning was, that I was not able to unexport my iscsi-exports with targetcli before rebooting, because targetcli tries to modprobge on each call and fails because the modules are already removed (I know, this is a bad thing in targetcli too) ...

So on arch this means for me stopping most services, do the kernel-update and reboot. This might work fine with some few hosts, but we run some hundreds of hosts maintained via puppet. On all other distros, we upgrade the kernels automatically as soon as our QA acknowledges the packet and the kernel becomes active as soon as the machine is rebooted anyway (in a maintenance window). And as explained above this is not possible on arch hosts especially as it is impossible to maintain targetd via targetcli anymore.

BR
Jens

tomk · 2014-10-29 10:58:20

So to summarise - your actual problem is that Arch is different from all other distros, and therefore has to be handled in a different way. What a surprise!

Presumably, there are aspects of Arch that you like, so it is up to you to decide if those are important enough to warrant changing your procedures to accommodate Arch. Assuming those maintenance windows of yours are scheduled in advance, couldn't the kernel upgrade be one of the maintenance tasks?

Lorddusty · 2014-10-29 11:41:16

tomk wrote:

So to summarise - your actual problem is that Arch is different from all other distros, and therefore has to be handled in a different way. What a surprise!
Presumably, there are aspects of Arch that you like, so it is up to you to decide if those are important enough to warrant changing your procedures to accommodate Arch. Assuming those maintenance windows of yours are scheduled in advance, couldn't the kernel upgrade be one of the maintenance tasks?

Well that's not quiet hitting the point. It is not, that it is different to other distros, but it's bad to remove stuff like kernel-modules as long as they are / might be needed to run the system stable! Removing a kernel or kernel-modules as long as the kernel is currently bootet and running is not fun ... making it impossible to properly and cleanly reboot after Kernel-Updates is not a feature!

Would you unplug your Memory from your mainboard while the system is running, when you're planning to put bigger modules into the machine?

drcouzelis · 2014-10-29 12:05:41

I find this idea fascinating to think about. I agree that it feels like something should be done differently, but as can be seen from that bug report, no one can come to a consensus even after five years of consideration.

I'm curious, how would you resolve the issue in Arch Linux?

Lorddusty · 2014-10-29 12:31:04

Well, one idea could be, that a kernel-update cleans up all older kernels except the running one (which e.g. could be found by checking with uname). But that would mean pacman might need some enhancements, e.g. the ability to pin packages to specific versions and especially multi-slotting or at least give the user the chance to decide if he wants to remove everything belonging to (specified) older packages on update (e.g. by setting parameters in pacman.conf).

Several other packagers already behaved like pacman in the past, but most of them evolved sooner or later to provide possibility to allow version-pinning or multi-slot installs... or they more or less disappeared, because they were not usable for production.

Actually I have the impression, that ArchLinux has a very good approach in lot's ot things, so why not thinking about evolving it further to become "production-environment-ready". And with "production-ready" I do not mean for a single system, where it doesn't matter if it is offline for half a day, but mass-systems which should be online almost 24/7 with a tollerance >=99.5%.

I think ArchLinux is not intended to be a "geek-only" distro, for giving geeks something to play with!

Trilby · 2014-10-29 12:38:59

I don't know how crazy this would be - but perhaps in the pre_upgrade script, the kernel package could move/copy all the existing module files into a temporary folder ( /tmp/current-modules/ ) then if there were a way to get the running kernel to look there for modules, then we'd be all set.

This would keep the modules around, but they'd be removed at reboot as they are in tmpfs.

tomk · 2014-10-29 12:54:19

Lorddusty wrote:

Well that's not quiet hitting the point.

But that's exactly the point you're making - you do certain things with all other distros, and you're saying it's impossible to do the same things with Arch.

Lorddusty wrote:

it's bad to remove stuff like kernel-modules as long as they are / might be needed to run the system stable!

I agree it's bad, but you have decided to do that - nobody's forcing you.

Lorddusty wrote:

making it impossible to properly and cleanly reboot after Kernel-Updates is not a feature!

Now you're just exaggerating. OK, if you insist on doing things that you know will not work, then sure, you make it impossible to cleanly reboot etc etc. But again, it's your decision - there is no ambiguity about the way Arch currently handles kernel upgrades, so your choice of Arch should be accompanied by an appropriate set of procedures.

I've yet to see any explanation why it's so hard to delay kernel upgrades until the time is right to deal with them. I do it all the time.

Lone_Wolf · 2014-10-29 16:20:47

If i understand correctly, modules that are loaded in memory stay available to the running kernel .

That suggests a simple workaround :
make sure all modules you need are loaded before running the kernel update.

ewaller · 2014-10-29 18:32:37

I will say that, having come here from Gentoo, I always had more than one kernel installed. When a new kernel came along, you would build it and its modules and would install them along side the other module(s). The boot partition would have multiple modules from which to select at boot time. When you reboot and try your shiney new module only to meet with epic failure, it was a simple matter to fall back and boot from the old module to fix the issue.

The point being, it is not an uncommon use case. For versioned releases such as Fedora or Ubuntu, maybe not so much. Gentoo is a rolling release and it uses this model.

Last edited by ewaller (2014-10-29 18:32:58)

Lorddusty · 2014-10-30 00:59:52

Lone_Wolf wrote:

If i understand correctly, modules that are loaded in memory stay available to the running kernel .
That suggests a simple workaround :
make sure all modules you need are loaded before running the kernel update.

This easyls works for most types of modules, but what with modules like e.g. configfs which are loaded by tools to communicate from userspace with the kernel, e.g. for properly unmounting remote-storages via infiniband ...

@tomk: So you tell me to umount the remote-storage holding the modules before I upgrade the kernel, because removing the modules means it is no longer possible to cleanly umount the disk after the upgrade and so run into the risk of fs-issues? If you tell me how I get the modules correctly installed on a not mounted disk, I will try to implement it... and if Arch is a "single-installed-kernel"-distro, why are grub and other multi-kernel-boot-loader available?

tomk · 2014-10-30 07:25:04

Lorddusty wrote:

@tomk: So you tell me to ... <snip>

I'm telling you to develop appropriate procedures to allow you to use Arch in your particular circumstances - I'm not concerned about the details, that's your area.

Lorddusty wrote:

if Arch is a "single-installed-kernel"-distro, why are grub and other multi-kernel-boot-loader available?

Is this a serious question? You can install as many kernels as you want - Arch provides two in the official repos - and boot-loaders are available so that people can boot their systems

Lorddusty · 2014-10-30 08:50:16

tomk wrote:

I'm telling you to develop appropriate procedures to allow you to use Arch in your particular circumstances - I'm not concerned about the details, that's your area.

Your point of "Works for me, so everything is fine" is quiet ridiculous. Face the fact: Removing stuff from filesystem while it is in use is simply wrong! Couple of years ago I read a read a nice guideline/presentation about kernel development an kernel upgrade procedures. It clearly stated that you should never remove kernel modules of a kernel currently in use. It's hilarious to keep the position "no need to change" for something which is obviously against all best practice and against all logic. I am sure you won't change the steering-wheel of your car while you are driving on the high-way.

It's nice to have a USP in a product, but the USP should not be doing things wrong. Had couple of discussions with several package-maintainers of Gentoo/Debian/Ubuntu/SuSE/Fedora about features in the past where the other side initially also held the position "we are not going to change this, it is a corner-case you are asking for", but this was always for feature-enhancements.

And yes, we have a workaround for this faulty behaviour, we are no longer using the original kernel-packages but build our own kernel-packages which are not screw things up on update.

tomk wrote:

Is this a serious question? You can install as many kernels as you want - Arch provides two in the official repos - and boot-loaders are available so that people can boot their systems

For booting a single simple linux-kernel you don't need the big overhead of a fat boot-loader like grub ... anyway, does not seem to make sense to discuss bugs with people who are not willing to enhance stuff by removing bugs instead on insisting on the position that they are never doing anything wrong.

I do not see, why it should be a problem to make a tuneable flag in config, which gives the user a chance to keep influence if an upgrade removes the older version completely or to keep the stuff. It might even hold the default value "Remove everything, don't care if it is in use" but could be changed to "keep old stuff unless I remove it manually". And if you don't want to put it into the config, why not creating an additional command-line-parameter "--keep-old" or something. This wouldn't change anything for people with simple systems who are happy with the way things are currently handled, and gives the opportunity to keep modules on disk for people who fear to change the steering-wheel while driving.

progandy · 2014-10-30 09:01:47

I'm thinking about a cleaner way to manage the kernel upgrade and got to this:
- split linux in linux-kernel and linux-modules-${KVER}
- linux-kernel depends on linux-modules-${KVER}
- build a service that searches for unneeded modules packages and removes them this can run on shutdown directly before unmounting.

a --keep-old options for pacman is a bad idea. You'll get file conflicts as soon as two package version use the same files.

Last edited by progandy (2014-10-30 09:06:34)

drcouzelis · 2014-10-30 13:49:28

Lorddusty wrote:

if Arch is a "single-installed-kernel"-distro

Arch Linux isn't a "single-installed-kernel"-distro, it's a "single-kernel-package"-distro (and that's not even including the LTS kernel). You can create a PKGBUILD files and install as many kernel versions as you like.

I agree that there's probably a better and safer way to perform kernel upgrades in Arch Linux. But at the same time, I understand that Arch Linux has been using its current method of kernel upgrades for over a decade with extremely few issues.

Last edited by drcouzelis (2014-10-30 13:49:38)

Leonid.I · 2014-10-30 18:37:23

@tomk:
Stop being silly and try to listen to others. There is clearly a problem; if you haven't encountered it, then your setup is not sophisticated enough.

@Lorddusty:
I think there are multiple things that people confuse when discussing this issue with Archlinux kernels (see the bugreport that WorMzy linked to), so let me try to explain:
1. Stability of the new kernel.
2. Ability to keep working after a kernel update.
3. Ability to _cleanly_ boot into a new kernel.

It's not like Arch devs don't know about these issues (they have for a long time). My guess why this is a low priority is because there are multiple issues with retaining the old kernel, the main being user support: (i) do we support booting a new userspace with an old kernel? In versioned distros like Fedora this is not an issue because userspace is not actively updated, but with Arch it is; (ii) for how long can we run-and-update the system w/o reboot before we start getting strange issues with userspace and kernel getting out-of-sync? Another possible reason is that Arch was not designed to be run by dumb sysadmins in a large company. So when there is a problem, users typically just build their own kernel, like you did.

So, the issue (1) above can be solved by having a linux-lts available. It is tested and guaranteed to work with an up-to-date system. The issue (2) is an invitation for a disaster, imho. It is best if the kernel update is followed by a prompt reboot. Finally, issue (3) is what you seem to be mainly concerned about.

Can you elaborate on how you build your kernel packages? I guess they provide=linux, no? Or you don't even have a linux packages installed, but instead e.g. linux-3.1, linux-3.2, etc.? How do you deal with pkgrels (linux-3.17.1-5)?

This is clearly a pacman issue, so why not just hack into pacman DB instead of building your own kernel (not that the latter is difficult)? For example, in /var/lib/pacman/local/ you do: mv linux-3.17.1 linux-old-3.17.1, replace a couple of strings in the files inside this dir, rm /boot/vmlinuz, and then pacman -S linux? As a result, you'll have 2 kernel packages: linux and linux-old, and avoid file conflicts...

Another solution would be for the kernel to read modules from several dirs, like /lib/modules/... and /run/modules/... Then you'll simply rsync /lib/modules to /run/modules on-boot.

EDIT: Oops, overlooked Trilby's post -- sorry.

Last edited by Leonid.I (2014-10-30 18:41:38)

olive · 2014-10-30 19:39:42

There are a lot of other things that can be removed while it's in use when you do an upgrade: for example: systemd, glibc, udev, xorg (although you can turn it off when doing an upgrade). Other distros usually keep the original kernel but not all of these other stuff. A bug in these other packages could render the system unbootable too, just like the kernel. Should arch adopt the same positions as other distros and make an exception for the kernel? I don't think so. Upgrading these other stuffs require rebooting too to have a clean system. By the way any other distros as well as Windows requires a reboot after upgrading system stuff.

Last edited by olive (2014-10-30 19:40:35)

Lorddusty · 2014-10-30 19:49:15

@Leonid.I: Thank you for your constructive comment! Nice to see that there ARE people on this board who care!

We actually did not build our own package, as we did not find a way yet how to make pacman not deleting the file. Instead we created a script which is triggered by our puppet.
This script does the following:

1.) check if kernel-files newer then the running one exist. If so check if there is already a copy of this files in a backup directory, if not copy the files belonging to that kernel into the backup directory
2.) check if files belonging to the running kernel are missing, if so, copy them back from backup-dir
3.) check if there are files which belong to an older version then the running kernel and the last one (either in original file locations or in backup-dir) exist, if so clean them up

For /boot/vmlinuz we move it to /boot/vmlinuz-$KVER and make a symlink to /boot/vmlinuz, same for initramfs. When step 2 from above is true, the symlink vmlinuz-old is pointed to the "current running version". This is also the pointer we use for step 3.

Hope I was able to explain it more or less understandable.

I am completely with you on your concerns. And the explanation for the "rolling distro" is also pretty clear, maybe you might want to sneak a little on how gentoo handles that. They call ist "Sloting". This is less a question of packaging the kernel but an issue for the pacman installer. The kernel-packages are always named with their full version-number in /boot and grub-config gets additional entries popped at the beginning when a new kernel is being installed (in gentoo genkernel-package does it like that). The modules are installed in a separate directory with their version-name so there are no file-conflicts.

Only issue which persists is how to clean up stuff. In gentoo this must be done manually. A nice approach is the one debian/ubuntu uses with having an option "--auto-remove" to the package-installer.

Leonid.I · 2014-10-30 23:05:09

@olive:
Yes, updating the running system is always risky (I've seem a discussion about this either on debian-devel or fedora-devel, but can't find a link atm), and it's true that kernel is just another package, so singling it out may not be so KISS... Also, I am personally against pacman doing some magic with my syslinux.cfg after a kernel upgrade (like inserting new entries, etc.).

EDIT: And, thatnks for the killing the myth that linux doesn't need to be restarted after an update
EDIT2: An unbootable system is not a concern, we are talking about an un-rebootable system.

@Lorddusty:
I don't get it. So, you manually unpack the Arch kernel package and just copy files? What about the pacman DB?

After thinking a bit more, what I said before is too compIicated. I would address the problem in two ways (both untested).

First, build versioned kernel packages like it was already discussed in the above bugreport (#16702).

Another way, if you'd rather not package/recompile things is to add some hackery to the -ARCH kernel update procedure, along the lines of what Trilby and Sergej said in the bugreport. Have you thought of something like this:
* Before a kernel update: mount tmpfs over /usr/lib/modules and bind-mount /usr/lib/modules somewhere (say /run/usr-lib-modules). The /usr/lib/modules is now empty and all the old modules can be accessed in /run/usr-lib-modules.
* rsync/cp -a modules dir (e.g. 3.17.1-1-ARCH/) to the tmpfs from underneath the mount point.
* Run an update what will replace modules in the tmpfs, but not on-disk.
* rsync/cp -a new modules to disk (from /usr/lib/modules to /run/usr-lib-modules) and move the old modules to the tmpfs.

The old kernel is still in the memory and will find its modules, so you'll be able to load them. Upon restart, the old modules are lost and you boot to a new kernel (hopefully). You can easily script the above procedure (I do something similar to relocate /var/spool/mail to RAM on-boot to avoid unnecessary writes to the SSD). Of course, now kernel (linux package) is special and all other updates should run with --ignore linux. Moreover, pacman -Qk run against linux will complain about missing files. But at least pacman DB is always consistent except that small time windows before restart.

Also note, that there is a discussion of how to make atomic directory renames/moves, but the above procedure is not atomic. So, if something reads your /usr/lib/modules while you are relocating it, this something is going to be confused. And surely, none of the above is acceptable as a distribution policy, but can be used in special setups like yours.

HTH.

Last edited by Leonid.I (2014-10-30 23:11:26)

Arch Linux

#1 2014-10-24 11:00:05

Autoremoval of old kernel-stuff on kernel update

#2 2014-10-24 12:12:27

Re: Autoremoval of old kernel-stuff on kernel update

#3 2014-10-24 12:18:57

Re: Autoremoval of old kernel-stuff on kernel update

#4 2014-10-24 19:36:14

Re: Autoremoval of old kernel-stuff on kernel update

#5 2014-10-29 10:58:20

Re: Autoremoval of old kernel-stuff on kernel update

#6 2014-10-29 11:41:16

Re: Autoremoval of old kernel-stuff on kernel update

#7 2014-10-29 12:05:41

Re: Autoremoval of old kernel-stuff on kernel update

#8 2014-10-29 12:31:04

Re: Autoremoval of old kernel-stuff on kernel update

#9 2014-10-29 12:38:59

Re: Autoremoval of old kernel-stuff on kernel update

#10 2014-10-29 12:54:19

Re: Autoremoval of old kernel-stuff on kernel update

#11 2014-10-29 16:20:47

Re: Autoremoval of old kernel-stuff on kernel update

#12 2014-10-29 18:32:37

Re: Autoremoval of old kernel-stuff on kernel update

#13 2014-10-30 00:59:52

Re: Autoremoval of old kernel-stuff on kernel update

#14 2014-10-30 07:25:04

Re: Autoremoval of old kernel-stuff on kernel update

#15 2014-10-30 08:50:16

Re: Autoremoval of old kernel-stuff on kernel update

#16 2014-10-30 09:01:47

Re: Autoremoval of old kernel-stuff on kernel update

#17 2014-10-30 13:49:28

Re: Autoremoval of old kernel-stuff on kernel update

#18 2014-10-30 18:37:23

Re: Autoremoval of old kernel-stuff on kernel update

#19 2014-10-30 19:39:42

Re: Autoremoval of old kernel-stuff on kernel update

#20 2014-10-30 19:49:15

Re: Autoremoval of old kernel-stuff on kernel update

#21 2014-10-30 23:05:09

Re: Autoremoval of old kernel-stuff on kernel update

Board footer