[SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

securetodeath · 2020-06-26 15:31:50

I'm thinking about implementing automatic/unattended upgrades only when it can be done safely.
Yes, I know that Arch is user centric and that I should be responsible for updating my system, but in the majority of cases I don't really have to intervene at all, and I would like to automate precisely these cases, since I want to update frequently for security.

The plan is to detect when manual intervention is required, and if so skip the automated upgrade and send an email to myself to do it manually. Questions:

1. Is there an existing reliable way to check if manual intervention may be needed?
I know there are scripts that check the Arch news, will this be sufficient, or should I be worried about pacnew files too? other things?
2. Assuming the aforementioned detection logic has a false negative, what are the risks?
For example, I know from previous Arch news that sometimes you must use `--overwrite`, or otherwise the upgrade will fail. But that doesn't seem like a big risk, because if the upgrade fails my
system should still be usable (assuming the upgrade is transactional), so I can later run that upgrade manually. So, I'm wondering if there are cases where blindly running `pacman -Syu` can
result in a broken system.
3. Assuming I broke my system with a blind `pacman -Syu`, is there a chance I will break it so badly that I won't be able to easily fix it from a live ISO?

Thanks in advance!

Last edited by securetodeath (2020-06-27 06:57:42)

Scimmia · 2020-06-26 15:39:43

1) .pacnew files, post-install messages, optional deps, etc. Pacman output is meant to be read by the sysadmin.
2) yes, if you don't take action on any of the items listed in #1, it can result in things being broken.
3) No.

Trilby · 2020-06-26 15:49:12

securetodeath wrote:

1. Is there an existing reliable way to check if manual intervention may be needed?

Yes, watch the output from pacman. I suspect you want to know if there is any other method. And the answer is no.

securetodeath wrote:

2. Assuming the aforementioned detection logic has a false negative, what are the risks?

We could provide examples including potential partial upgrades and database locks, but they would only be a subset of the potential problems. Do not try to enumerate badness; you will fail.

securetodeath wrote:

3. Assuming I broke my system with a blind `pacman -Syu`, is there a chance I will break it so badly that I won't be able to easily fix it from a live ISO?

There is no way to know what would be easy or hard for you. Is there a chance you could break something that could not be fixed from a live ISO: no. Would it be easy for you to do so ... there's no way to know.

securetodeath · 2020-06-26 16:10:13

Scimmia wrote:

1) .pacnew files, post-install messages, optional deps, etc. Pacman output is meant to be read by the sysadmin.
2) yes, if you don't take action on any of the items listed in #1, it can result in things being broken.
3) No.

Here's my understanding:
- pacnew files can break the system when a package is not backwards compatible with the user edited file
- post-install messages: I guess that's also something that could be critical
- new optional deps: could this lead to breakage?

Scimmia · 2020-06-26 16:20:32

If you're using a feature that needs that new optional dep, sure.

securetodeath · 2020-06-26 16:25:28

Trilby wrote:

securetodeath wrote:
1. Is there an existing reliable way to check if manual intervention may be needed?
Yes, watch the output from pacman. I suspect you want to know if there is any other method. And the answer is no.
securetodeath wrote:
2. Assuming the aforementioned detection logic has a false negative, what are the risks?
We could provide examples including potential partial upgrades and database locks, but they would only be a subset of the potential problems. Do not try to enumerate badness; you will fail.

I know there's no silver bullet here, but I'm looking for a solution that will run unattended for most upgrades, usually without breaking my system.
I'm willing to deal with breakage once in a while (but not data corruption or something very nasty). I will also mitigate any breakage by using something like LVM/ZFS snapshots before upgrades, regular backups, etc.

BTW, I would argue that automating upgrades that don't require manual intervention can actually reduce the chance of a broken system for some users. The reason is that manually looking at pacman's output when it almost never outputs anything actionable (which is a good thing, of course) can lead users to ignore that output. Maybe that's not an issue to most users, but it certainly is for me.

securetodeath · 2020-06-26 16:29:40

Backing off slightly: is there a way to do a dry run of pacman so that I can parse its output and see if there's anything that requires attention, without actually doing the upgrade?
I guess I could use build some sort of temporary COW filesystem based on the real root filesystem and run pacman with the `--sysroot` parameter, but I'd prefer to use an existing solution if there is one.

eschwartz · 2020-06-26 17:49:30

There's no way to mock pacman and make it run post_upgrade scripts to see if there are warnings.

However, I see no reason why you couldn't clone the database, open a handle on the new copy, and do pacman -Sy (this is what checkupdates does), THEN enumerate the files which would be installed/updated/or removed by the update and check for conflicts. You could also compare the before/after list of optdepends.

You will need to write all this yourself however, since it currently doesn't exist. Perhaps as a custom program linked to libalpm.

finoderi · 2020-06-27 05:22:05

https://deviq.com/yagni/ And there is KISS principle as well.

securetodeath · 2020-06-27 06:54:42

eschwartz wrote:

There's no way to mock pacman and make it run post_upgrade scripts to see if there are warnings.
However, I see no reason why you couldn't clone the database, open a handle on the new copy, and do pacman -Sy (this is what checkupdates does), THEN enumerate the files which would be installed/updated/or removed by the update and check for conflicts. You could also compare the before/after list of optdepends.
You will need to write all this yourself however, since it currently doesn't exist. Perhaps as a custom program linked to libalpm.

Thanks, that's very helpful!
Do you think this will be simpler to do than something based on fileystem snapshots, OSTree, etc? My thinking is to do a COW clone of root, run pacman on the mount point, and check it's output.

Last edited by securetodeath (2020-06-27 07:27:15)

securetodeath · 2020-06-27 06:56:52

finoderi wrote:

https://deviq.com/yagni/ And there is KISS principle as well.

That doesn't add anything to the discussion.
I explained pretty clearly in my original post why I want to do this.
If you do have anything substantive to say I'd be happy to hear it though.

Skunky · 2020-06-27 07:23:36

I wonder if there is a proper way to make pacman hide installing $package message, so you can only see more useful output

finoderi · 2020-06-27 07:27:00

I think there are pretty good reasons why this feature still doesn't exist.
I just don't like the general direction of this... research. 'Let's not reboot the system', 'let's not upgrade the kernel because of that', 'let's try to perform automatic updates when we can get away with that'... Something doesn't feel right...

Last edited by finoderi (2020-06-27 07:42:06)

Awebb · 2020-06-27 08:26:02

Wow. The thread title explicitly asks for risks of unattended upgrades. OP is aware, that there are problems and wants to get community input to see if it's a dumb idea to begin with. If "there are pretty good reasons why this feature still doesn't exist", then be a dear and list them. Other package management systems have robust unattended updates, so "why doesn't pacman/Arch" is a legitimate question.

If this thread goes against some sort of doctrine or mantra you have imposed on yourself, then this is the only thing that doesn't feel right here.

securetodeath · 2020-06-27 10:18:36

finoderi wrote:

I think there are pretty good reasons why this feature still doesn't exist.
I just don't like the general direction of this... research. 'Let's not reboot the system', 'let's not upgrade the kernel because of that', 'let's try to perform automatic updates when we can get away with that'... Something doesn't feel right...

Again, this doesn't add anything to the discussion. Actually, it negatively impacts the discussion.
If there are "good reasons", you should list them, as it's not clear what those reasons are.

And whether or not you "like" this research isn't relevant at all. If you have sound technical arguments for why this idea can't be implemented (because, for example, it contradicts the design of the system
too much), feel free to share them. Otherwise, refrain from making comments that don't advance the discussion.

eschwartz · 2020-06-28 03:55:20

securetodeath wrote:

eschwartz wrote:
There's no way to mock pacman and make it run post_upgrade scripts to see if there are warnings.
However, I see no reason why you couldn't clone the database, open a handle on the new copy, and do pacman -Sy (this is what checkupdates does), THEN enumerate the files which would be installed/updated/or removed by the update and check for conflicts. You could also compare the before/after list of optdepends.
You will need to write all this yourself however, since it currently doesn't exist. Perhaps as a custom program linked to libalpm.
Thanks, that's very helpful!
Do you think this will be simpler to do than something based on fileystem snapshots, OSTree, etc? My thinking is to do a COW clone of root, run pacman on the mount point, and check it's output.

Filesystem snapshots will obviously be much easier, albeit use more resources/disk IO. They will also show post_upgrade script messages.

You "could" write your own custom programs to save disk IO at the expense of scriptlet messages, doesn't necessarily mean you "should".

Skunky wrote:

I wonder if there is a proper way to make pacman hide installing $package message, so you can only see more useful output

... no? I mean, you could write your own libalpm client to do whatever you want. But pacman certainly cannot and will not. pacman is not optimized for scripted use. The closest it comes is --noconfirm, but users are still intended to view its output.

You could, however, filter its output. Alternatively check in pacman.log for anything logged as [ALPM-SCRIPTLET], which covers hook or script output, but not messages about new optdepends.

Skunky · 2020-06-28 08:36:17

Thanks for your reply, so reading [ALPM-SCRIPTLET] messages would be enough to know if there is manual intervention needed? When installing/updating alot of packages there can be alot of "installing $packagename" and one could miss some critical message that ask for manual intervention, that's why i tought if installing packages messages could be hidden with some kind of less verbose mode

cirrus · 2020-06-28 11:41:20

Though I too would advise against this, maybe informant pkg would help,it halts update if any unread news items. Like most informants it can be a nuisance.

eschwartz · 2020-06-28 13:22:34

cirrus wrote:

Though I too would advise against this, maybe informant pkg would help,it halts update if any unread news items. Like most informants it can be a nuisance.

What is an "informant pkg"?

Head_on_a_Stick · 2020-06-28 13:27:49

Just use btrfs and snapper to take snapshots before each upgrade then boot into the snapshot if there are any problems:

https://wiki.archlinux.org/index.php/Sn … _snapshots

securetodeath · 2020-06-28 13:52:09

eschwartz wrote:

securetodeath wrote:
eschwartz wrote:
There's no way to mock pacman and make it run post_upgrade scripts to see if there are warnings.
However, I see no reason why you couldn't clone the database, open a handle on the new copy, and do pacman -Sy (this is what checkupdates does), THEN enumerate the files which would be installed/updated/or removed by the update and check for conflicts. You could also compare the before/after list of optdepends.
You will need to write all this yourself however, since it currently doesn't exist. Perhaps as a custom program linked to libalpm.
Thanks, that's very helpful!
Do you think this will be simpler to do than something based on fileystem snapshots, OSTree, etc? My thinking is to do a COW clone of root, run pacman on the mount point, and check it's output.
Filesystem snapshots will obviously be much easier, albeit use more resources/disk IO. They will also show post_upgrade script messages.
You "could" write your own custom programs to save disk IO at the expense of scriptlet messages, doesn't necessarily mean you "should".

Thanks, it sounds like what could be relatively easy and safe is to do the upgrade in a COW clone of the filesystem, and only switch to the new filesystem if nothing unexpected appears in Pacman's output (meaning only regular package upgrade messages). I'm not sure if this is actually easy though, I know for example that Fedora Silverblue (based on OSTree) requires to reboot in order to apply changes to the base system packages/files.

securetodeath · 2020-06-28 13:52:41

eschwartz wrote:

cirrus wrote:
Though I too would advise against this, maybe informant pkg would help,it halts update if any unread news items. Like most informants it can be a nuisance.
What is an "informant pkg"?

Just a guess that the reference is to: https://github.com/bradford-smith94/informant

securetodeath · 2020-06-28 13:57:32

Head_on_a_Stick wrote:

Just use btrfs and snapper to take snapshots before each upgrade then boot into the snapshot if there are any problems:
https://wiki.archlinux.org/index.php/Sn … _snapshots

Thanks for the reference, and see my previous posts: that's indeed one of the leading directions. Ideally I would be able to verify that there's no special messages from pacman before changing the mounted filesystem, and only switch if the upgrade is deemed safe. Otherwise, I'd prefer to be notified to check the output manually.

eschwartz · 2020-06-28 14:45:55

securetodeath wrote:

eschwartz wrote:
cirrus wrote:
Though I too would advise against this, maybe informant pkg would help,it halts update if any unread news items. Like most informants it can be a nuisance.
What is an "informant pkg"?
Just a guess that the reference is to: https://github.com/bradford-smith94/informant

That software scares me. Why is it using a pacman hook to force pacman -Syu to fail after having done the -y part but not the -u part? There are other programs which do something similar, but only by wrapping pacman and checking if you read the news before finally running pacman -Syu.

This software on the other hand is vulnerable to partial updates, by default/design.

2ManyDogs · 2020-06-28 15:13:32

Why is this topic marked [SOLVED]? It appears to be an ongoing "discussion".

Arch Linux

#1 2020-06-26 15:31:50

[SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#2 2020-06-26 15:39:43

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#3 2020-06-26 15:49:12

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#4 2020-06-26 16:10:13

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#5 2020-06-26 16:20:32

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#6 2020-06-26 16:25:28

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#7 2020-06-26 16:29:40

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#8 2020-06-26 17:49:30

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#9 2020-06-27 05:22:05

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#10 2020-06-27 06:54:42

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#11 2020-06-27 06:56:52

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#12 2020-06-27 07:23:36

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#13 2020-06-27 07:27:00

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#14 2020-06-27 08:26:02

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#15 2020-06-27 10:18:36

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#16 2020-06-28 03:55:20

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#17 2020-06-28 08:36:17

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#18 2020-06-28 11:41:20

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#19 2020-06-28 13:22:34

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#20 2020-06-28 13:27:49

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#21 2020-06-28 13:52:09

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#22 2020-06-28 13:52:41

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#23 2020-06-28 13:57:32

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#24 2020-06-28 14:45:55

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

#25 2020-06-28 15:13:32

Re: [SOLVED] Risks of automatic/unattended upgrades (w/o manual work)

Board footer