[SOLVED] How to update a system which hasn't been updated in a year?

pgoetz · 2017-02-23 18:21:09

I run a production server which for various reasons hasn't been updated in a year (but now needs to be updated). I created a mirror test server and ran

# pacman -Sy archlinux-keyring && pacman -Syyu

and the test server appeared to upgrade successfully, however I still have some concerns about the major pacman change (adding transactional hooks) that occurred during this time (as described here).

For example, I did have to run

# mkinitcpio -p linux

by hand after the upgrade, as the system no longer recognized the root partition on reboot after the upgrade. I'm concerned that there might be other packages which have coverted to using transactional hooks resulting in partially compromised package upgrades.

So, I'm wondering if, for example, the solution to this is to upgrade pacman by hand first (i.e. download package, run pacman -U pacman*) before attempting a full system upgrade.

Last edited by pgoetz (2017-02-23 20:19:33)

Awebb · 2017-02-23 18:33:54

Re-install lite: The arch installer is your friend. Make a backup, boot, arch-chroot, uninstall everything and pacstrap all former explicitly installed packages, pulling in all the dependencies, then manually build your AUR packages. Merge /etc from the backup, use a diff tool if you are uncertain.

- or -

Use pacman to manually re-install all the packages to trigger the hooks. Make sure you retain the dependency status in some way.

- or -

If I'm not entirely mistaken, the hooks are stored in "/usr/share/libalpm/hooks", pacman -Qo is your friend here. Again, make sure to be clever about --asdeps.

EDIT: Just tried some "for i in /usr/share/libalpm/hooks/*...", it really seems to give you a list of all packages using hooks.

Here, for the vultures to pick on my bash:

pacman -Qqo /usr/share/libalpm/hooks/* | uniq | LANG=C xargs pacman -Qi | grep -E "Name|Install Reason"

Last edited by Awebb (2017-02-23 18:47:54)

Trilby · 2017-02-23 19:04:02

Vulture incoming: rather than using grep, I'd just ask pacman for those that were explicitly installed or not:

pacman -Qeq $(pacman -Qqo /usr/share/libalpm/hooks/* | uniq)
pacman -Qdq $(pacman -Qqo /usr/share/libalpm/hooks/* | uniq)

That way the lists can easily be passed as they are to another command. However, unless I'm missing something, the install reason is irrelevant. If you reinstall a package, the install reason doesn't change. So the following would be sufficient:

pacman -S $(pacman -Qqo /usr/share/libalpm/hooks/* | uniq)

And additionally, pacman is smart enough to not try to install the same package twice in the same transaction, so the uniq is not needed:

pacman -S $(pacman -Qqo /usr/share/libalpm/hooks/*)

Awebb · 2017-02-23 19:09:15

Did I just grep instead of -Qqe/-Qqd? Major Facepalm, reporting for duty. Time to go to bed.

pgoetz · 2017-02-23 19:11:41

Awebb wrote:

Use pacman to manually re-install all the packages to trigger the hooks. Make sure you retain the dependency status in some way.

So, reinstalling everything is precisely the scenario I'd prefer to avoid, since it is a production server, so I'd like it to be offline for as little a time as possible.

Thanks for the trick regarding examining the hooks. However, these presumably won't be there prior to the upgrade (since hooks didn't exist then). I think your suggestion here is to do the upgrade and then look for packages which use hooks in /usr/share/libalpm/hooks.

This leaves me with the quoted suggestion, which I'm not following. Your suggestion is to re-install every package on the system regardless of whether or not it's required by the update? Why would that be necessary? Also, i'm not sure what "retain the dependency status in some way" means. Aren't the dependencies going to be hauled in automatically by the packaging system?

Thanks.

Awebb · 2017-02-23 19:18:22

Okay, I understand the confusion. My third recommendation was to update the system, fix mkinitcpio, reboot, identify the packages with a hook and re-install those packages. Then all you need to do is merge the pacnew files and be done with it.

About dependencies: If you install a package with "pacman -S $package", then the install reason will be set to "explicitly installed" as opposed to "installed as a dependency of another package". Simply doing so breaks automatic dependency handling in case you want to remove a tree of packages. See Trilby's post and man pacman regarding -Qe, Qd and -D --asdeps.

pgoetz · 2017-02-23 19:20:12

Trilby wrote:

And additionally, pacman is smart enough to not try to install the same package twice in the same transaction, so the uniq is not needed:
pacman -S $(pacman -Qqo /usr/share/libalpm/hooks/*)

What I'm hearing is that I should be covered (modulo AUR packages, of which there are a few, but most are only loosely tied to the system version, e.g. cyrus imapd) if I do the following:

# pacman -Sy archlinux-keyring && pacman -Syyu
# mkinitcpio -p linux  <-- (this was required for my mirror test server)
# reboot
# pacman -S $(pacman -Qqo /usr/share/libalpm/hooks/*)

If correct, this would make for a very nice little Wiki page. I'm sure there are other naughty Archers who can't, won't, or don't upgrade their systems for a long time, but have established sufficient complexity to prefer to avoid a complete re-instal.

This settles it for me, though. I'm going implement a configuration management system (probably Ansible) in order to make reinstalls a no brainer. Don't want to go through this anxiety again, even if the system is fairly robustly backed up.

Update: So, I just ran

# pacman -Qqo /usr/share/libalpm/hooks/*

on a fully updated similar server and also on a (Mate GTK3) desktop, and hooks appear to be a much bigger issue on desktop systems. On my server it was mostly glibc, linux, and systemd:

[root@toad ~]# pacman -Qqo /usr/share/libalpm/hooks/*
linux
glib2
glib2
systemd
systemd
systemd
systemd
texinfo
texinfo
ca-certificates-utils
vim-runtime

Last edited by pgoetz (2017-02-23 19:26:29)

JohnBobSmith · 2017-02-23 19:26:02

I think what Awebb meant is that you wouldn't erase data and do a more literal re-install with a re-download, but rather run "pacman -S <software>" For example, if I do it to my kernel...

[jbs@dmb-gaming-laptop ~]$ sudo pacman -S linux-lts
warning: linux-lts-4.4.50-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) linux-lts-4.4.50-1

Total Installed Size:  77.60 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] y
....

Notice it says that everything is up to date and it wants to re-install. Accepting the prompt will make the hooks run...

...
:: Running post-transaction hooks...
(1/3) Install DKMS modules
(2/3) Updating linux-lts initcpios
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 4.4.50-1-lts
...

And all is well. This entire process took about 30 seconds, if that.

EDIT: Too slow, lots of people on at this hour.

Last edited by JohnBobSmith (2017-02-23 19:27:50)

pgoetz · 2017-02-23 19:29:56

JohnBobSmith wrote:

I think what Awebb meant is that you wouldn't erase data and do a more literal re-install with a re-download, but rather run "pacman -S <software>"

Got it -- Thanks.

But I think for this to work I would need to upgrade pacman by hand first (download new pacman; pacman -U pacman*.pkg*), assuming this is even possible. Hence my original question about whether or not it makes sense to upgrade just pacman before running a full system upgrade.

JohnBobSmith · 2017-02-23 19:59:33

I see no immediately obvious reasons why you couldn't use pacman to update pacman. I know for a fact that this is possible. The trick would be to update all of the other software pacman depends on, given your year's worth of absence.

...from the output of: pacman -Qi pacman
Depends On      : bash  glibc  libarchive  curl  gpgme  pacman-mirrorlist
                  archlinux-keyring
...

The trick is to do this without breaking too much else in the process. This seems impossible, given those dependencies. One other pesky file/library that I've also encountered during a partial upgrade fail[1] is libgcrypt. I see this leading to an inevitable trail of partial upgrades and partial breakages... Don't do this. Curious, if your upgrade worked from the installation side of things:

pgoetz wrote:

... I created a mirror test server and ran
# pacman -Sy archlinux-keyring && pacman -Syyu
and the test server appeared to upgrade successfully,
...

Why would you want to upgrade pacman separately in the first place? Other than, of course, to attempt possible solutions which have already been discussed and deemed unlikely to work. Unfortunately I don't have much more to add, so I'll leave it at that. But don't repeat my pacman mess with a similar (needing to re-install pacman) situation.
[1]:https://bbs.archlinux.org/viewtopic.php?id=214998

I hope I have been of some use.

Slithery · 2017-02-23 20:01:31

Another option is to use the ALA to update the system in steps, say one month at at time from when it was last updated.

I've successfully used this method to update a system that had seen no updates for about 3 years.

https://wiki.archlinux.org/index.php/Ar … cific_date

pgoetz · 2017-02-23 20:18:27

JohnBobSmith wrote:

But don't repeat my pacman mess with a similar (needing to re-install pacman) situation.
[1]:https://bbs.archlinux.org/viewtopic.php?id=214998
I hope I have been of some use.

I'm going to call this hugely helpful in terms of convincing me to not attempt to re-install pacman first.

Thanks, everyone!

I'm going to mark this issue as solved (and I learned a few things about pacman as an added bonus).

Trilby · 2017-02-23 20:47:34

Awebb wrote:

About dependencies: If you install a package with "pacman -S $package", then the install reason will be set to "explicitly installed" as opposed to "installed as a dependency of another package". Simply doing so breaks automatic dependency handling...

This is not true. I used to think this was the case too, but pacman is smarter than us. As noted in my previous post, if you use `pacman -S <pkg>` for an already installed package, it will be reinstalled (triggering any necessary hooks) but it will *not* change the installation reason.

pgoetz wrote:

What I'm hearing is that I should be covered... if I do the following

That is a correct interpretation of our suggestions. Whether our suggestions are correct or not remains to be seen. I *think* that would work just fine, but I have no direct experience to confirm.

pgoetz · 2017-02-23 21:08:22

Trilby wrote:

That is a correct interpretation of our suggestions. Whether our suggestions are correct or not remains to be seen. I *think* that would work just fine, but I have no direct experience to confirm.

I'll post a follow up for posterity's sake. Based on my experience with the mirror test server and the helpful suggestions everyone has made on this post, I'm quite confident this will go off without a hitch.

Also, thanks for confirming your observation that the installation reason doesn't change on reinstallation by pacman -S. So, then in those instances where you actually want to convert a former dependency into an explicit install (say because you're now using this dependency for something entirely different and don't want a tree uninstall to wreak havoc) it appears pacman already has us covered:

pacman -S --asexplicit  my_package

Trilby · 2017-02-23 21:27:56

There's no need to reinstall the package for that, just use the -D flag if the goal is only to change the installation reason.

Awebb · 2017-02-24 00:18:47

When did this change? I used to -S packages to change the state to explicit before I learned about -D. Good riddance, then!

eschwartz · 2017-02-24 02:25:29

Awebb wrote:

When did this change? I used to -S packages to change the state to explicit before I learned about -D. Good riddance, then!

In pacman 3.4.0, see FS#12950 and https://git.archlinux.org/pacman.git/co … 2c992f492f
This was mid-2010, by the way.

Awebb · 2017-02-24 08:03:40

Eschwartz wrote:

Awebb wrote:
When did this change? I used to -S packages to change the state to explicit before I learned about -D. Good riddance, then!
In pacman 3.4.0, see FS#12950 and https://git.archlinux.org/pacman.git/co … 2c992f492f
This was mid-2010, by the way.

I've been working around a fixed bug for seven years. Nice.

apg · 2017-02-24 13:33:47

Awebb wrote:

Eschwartz wrote:
Awebb wrote:
When did this change? I used to -S packages to change the state to explicit before I learned about -D. Good riddance, then!
In pacman 3.4.0, see FS#12950 and https://git.archlinux.org/pacman.git/co … 2c992f492f
This was mid-2010, by the way.
I've been working around a fixed bug for seven years. Nice.

That was the addition of the -D flag, -S has preserved the install reason since 2005.

Arch Linux

#1 2017-02-23 18:21:09

[SOLVED] How to update a system which hasn't been updated in a year?

#2 2017-02-23 18:33:54

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#3 2017-02-23 19:04:02

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#4 2017-02-23 19:09:15

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#5 2017-02-23 19:11:41

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#6 2017-02-23 19:18:22

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#7 2017-02-23 19:20:12

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#8 2017-02-23 19:26:02

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#9 2017-02-23 19:29:56

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#10 2017-02-23 19:59:33

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#11 2017-02-23 20:01:31

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#12 2017-02-23 20:18:27

Re: [SOLVED] How to update a system which hasn't been updated in a year?

Thanks, everyone!

#13 2017-02-23 20:47:34

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#14 2017-02-23 21:08:22

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#15 2017-02-23 21:27:56

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#16 2017-02-24 00:18:47

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#17 2017-02-24 02:25:29

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#18 2017-02-24 08:03:40

Re: [SOLVED] How to update a system which hasn't been updated in a year?

#19 2017-02-24 13:33:47

Re: [SOLVED] How to update a system which hasn't been updated in a year?

Board footer