Request for a "NeverUpgrade" option for pacman.

xirus · 2003-07-12 18:44:19

Yesterday something very serious happened, pacman moved my grub menu.lst file to menu.lst.pacsave. If I wouldn't have spotted this, my system would have become unbootable. This can be fixed by putting it in NoUpgrade (pacman.conf) by default, but there is an underlying problem here.

Pacman will always move your modified files (excluding the ones in NoUpgrade) to .pacsave and use a default file instead. Imagine this would happen on the httpd.conf (apache), my.cnf (mysql), php.ini (php),... files on a production server! It shouldn't, the server admin should have seen that pacman moved them, or he should have put them in the NoUpgrade line. But we're all human and these things happen. And a human error like this already happened with the grub file.

I've always been concerned with the actions of pacman. As Gyroplast said on irc: "Don't f*cking touch any already exiting config files until I explicitly tell you to!".

I've also noticed that a lot of times, these 'new' files are just the same as the original (unmodified) file, or are not worth the upgrade. It only costs time moving the old file back to its original position.

Newbies, people who didn't see that pacman moved their old file (on big updates, the console can even be to small to show this), people that do a "pacman -Syu" through cron,.. will never notice that their system can become unbootable, that apache's config file has been moved, so all their client's sites will be down after a reboot or apache restart,...

This is a very serious issue!

I have a very simpel solution though. I'm sure there are some people that are used to the old way and want to keep it, so we should give people the choice. Either use the old system, or the new. Let's say there is a "NeverUpdate" = yes/no option in pacman.conf. (default: YES!)

If you say no, everything will be back to 'normal'.

If you say yes, things will be different. We will now work with a whitelist instead of a blacklist as Gyroplast proposed. So instead of the "NoUpgrade" option, we will have an "Upgrade" option. All files will be protected by default (and use the .pacnew system), only the files in "Upgrade" can be overwritten, and moved to a .pacsave version. This is all VERY easy (and fast) to implement, if I knew C I would do it myself
I think it's important that this happens as soon as possible, to prevent things like the grub fiasco to happen again.

We can go further however: when a packager creates a new package with config files, he can tell pacman what has been changed (with a new option in PKGBUILD). If I upgrade that package, pacman will tell me exactly what has been added/changed so I can upgrade my file really quickly. Much faster than doing a 'diff' or just comparing it line by line. More work for 1 packager, but less work for hundreds of users. (and that 1 packager will also benefit from this, because he will save time updating packages from other packagers, so a win-win situation 8)). I realize that this will be much more work to implement, but would be a very nice feature in some future version of pacman.

I'm sure there will be some discussion about this, new ideas, people that hate me,..
I hope the Arch developers (particulary apeiro) will join this discussion, and also explain to us why the current pacman is so radical.

dariball · 2003-07-12 19:46:52

this is not as easy as u think.....

i think this is more complicated :
in a productive enviroment it's clear like u said, but if u are using the box just for a desktop for a more or less newbie uses the box then it would be nice to have files overwritten if i re-install a given package... or u willl re-install and re-install and the app won't work as long as u delete the file by hand or chnage it by hand, what isn't always that easy to know, especially for a linux noob

apeiro · 2003-07-12 20:43:14

Both are valid points... I copied rpm's decisions on this one, and it seemed to make sense at the time, in that it was the better of two non-perfect options.

I think the NeverUpdate option is a great idea, though I'll probably call it UpdateConfigs or something a little clearer. The real debate is what the default option should be.

Check out the behavior RPM takes and the rationales behind it, especially the last case: Original file = X, Current file = Y, New file = Z

andy · 2003-07-12 20:45:16

I think, I like =X¥®µ§='s idea. I always compare Arch to FreeBSD (which is quite legitimate, I think ;-) ... ). And I like both systems (who would have guessed ...). Anyways, coming to the point.

FreeBSD is quite similar. A system upgrade never touches any config files ... ever. Also, to catch up, there is this tool mergemaster. This is a very fancy diff. It is a shell script that shows the admin the diffs and then asks the admin what to do about each file. One can take the old, the new or even actually merge parts.

What I want to bring across is that this kind of strategy works for successful, production systems (FreeBSD), so it may be well worth for Arch.

And here is an idea for dariball. Instead of calling the option NeverUpdate, call it "UpdateStyle" and give options "SaveFirst" "SimpleFirst" or the like. And document of course that SaveFirst is for production systems where the system manager takes care himself of updating config files and that SimpleFirst is for those who don't want to think about every bit and let Arch do it.

Just my ideas ....

xirus · 2003-07-12 21:35:12

Ok, given the explanation on the RPM behaviour and dariball's post it's indeed correct that the new program might not work with the old config file.

So here we have some options.

1) We take the least damaging way, this would be the new way, since most new config files don't add anything new as I already posted. It will happen less that a program doesn't work because of an old config file, than the other way around - I think, I can't prove this of course.

But that would indeed give the problem of some programs not working because of an old config file, agreed. Imho it's still better than the old way though.

2) I've also used FreeBSD, and like andy, I think they have a near-perfect system. It's a bit like what I said earlier (which might be easier to program than fbsd's solution) -> before installing the program, you show the user what has changed in this file (problem: maybe that file already changed x times, so pacman should know what version the user has, I'm not sure if this can be done in a safe way? some files might also be too complicated to let pacman tell you what changed in a clean way) and give him the choice: ignore and install the package, don't install the package or change the config file and install the package.

3) Or just use something like the fbsd system, if the problems in (2) can't be solved. But this might require some programming to be done, unless you can somehow use their system. I believe Debian uses something similar as well.

This would mean that the old days of 'type pacman -Syu and my whole system will be updated without even looking at it' will be over. Security comes at a cost... But we can still give users the option between these things.

Btw dariball/apeiro, the problem that programs won't work with old config files is not a NEW problem. It already exists with the protected files now. I noticed that with the last update, mysql's config file changed (which is protected on my system) so it might not have worked after the update. But if I didn't protect it, mysql would also not work since it needs a username in the config file. Both ways give this problem.
Since Arch Linux is maturing a lot now, it might be a good time to consider all this (and maybe also the suggestion to let pacman download files from different mirrors at the same time, to save time - http://bbs.archlinux.org/viewtopic.php?p=4482#4482) and make a big pacman update with a freebsd config file system, and other things that would add value to pacman. No bloat of course
This doesn't make it newbie friendly, but that's the problem with having up do date software, arch is not a real newbie distro after all, it expects people to know how to work with config files, so I think this is a non-issue, is it?

What do you guys think?

Mork II · 2003-07-13 11:40:18

I like this idea, no matter what you call it (Still "UpdateConfigs: yes/no" is the most comprehensive, and no I'm not sucking up . Maybe not that much magic but easy to understand and use.

I would prefer "yes" as default, beeing a desktop user and all. In the default install a few files (lilo.conf, menu.lst and a few others) should be NoUpgrade. I do not remeber if it works this way already since it was fifty-three days since I last touched a linux box. (Abstinence? Who? Me?)

xirus · 2003-07-13 11:59:30

the name doesn't matter

the NeverUpgrade thing was just an example to explain what I meant

Wether it's named UpdateConfigs, Don'tF*ckingTouchMyFiles or YourMama, the only thing that matters is the system itself :mrgreen:

and if it would be called UpdateConfigs, I'm perfectly happy with it

Gyroplast · 2003-07-13 16:49:57

I have considered this issue for a while, and I have come to the conclusion that it'd be perfectly adequate to dump the whole NoUpdate option altogether, because of how pacman handles the XYX case (original and new file are identical, but the current file on disk differs). In this case, pacman will, by default and independent of any NoUpgrade settings, leave the existing file alone. This means that ONLY in the XYZ case, ie. all three files are different, pacman replaces the config file and saves the current one as a .pacsave. If that happens, though, the original file and the new file are different, which means that the default config you have been using as a base for the modified file has been changed! As these changes do never happen without a good reason, you are _forced_ to look at any file that creates a .pacsave anyway to be sure to not miss important changes. You'll notice that IF you set it as NoUpgrade, you'll effectively need to do the same thing to guarantee the config syntax has not changed that much to leave your current config unusable.

If you NoUpgrade a file, though, a .pacnew file will be _always_ generated, regardless if it differs from the original file or not.
So, in fact, it even helps to remove all Noupgrade Lines, since you cannot distinguish between a pacnew that does differ from the original file and a pacnew that does not.

If you switch all that crap off, you'll _only_ get .pacsave warnings if something _really_ has changed that will need your attention in every case, at the expense that a sudden restart may leave your service unavailable.

IMO, that's what these warnings are good for, so either consider this NoUpgrade behaviour a bug, or remove it entirely. The first option would be better, though, as it's sort of a safety net that's nice to have, but no admin should really let a package manager decide a sort of thing which it simply cannot decide in a correct way in all cases. It's too dangerous.

So, in the end, I suppose the existing implementation is after all sufficient. Your files won't be touched by default, even if you forget to add them to the NoUpgrade list, unless the config has changed since the last upgrade, in which case you WILL get a warning message. Can't help you if you don't read it. Noone and nothing can.

Greets,
Dennis

xirus · 2003-07-13 20:09:43

hmm now you're contradicting yourself a bit, unless I've misunderstood your post - first you say that you don't want any of your config files to be changed, and now you're saying that every config file should be changed :shock:

The fbsd system is basicly what you say, only it doesn't require you to look at the files, but demands you to do this while updating, with some sort of fancy diff program. This to prevent people from not seeing the warnings due to all kind of reasons (too much packages so it disappears from the console, cron job, accidently closing the xconsole window,...)

Gyroplast · 2003-07-13 20:55:14

Where am I saying I want every config file to be changed? My argumentation isn't as clear-cut as I wanted it to be, but basically my opinion has changed a bit since I actually fiddled around a bit with pacman.

I found that no config file will be overwritten under any circumstances as long as the new file is the same as the original file. Mind that "original file" != "the current file on disk".
The one and only possibility where pacman is overwriting an existing, modified config file (unmodified config files are no problem, as they are, well, unmodified. ) is the XYZ case, ie. the original file (the file that was installed by the last installation/update of the package), and current file on disk and the new file to be installed are all different. This means, however, that the packaged default configuration file has been changed. In most cases these changes are drastic and/or important, for example a new config file syntax has to be used, in which cases you have to intervene manually in either case, since your modified version will not work anymore, either. If only the lba32 option is added to lilo, which is clearly not a version change that would leave the modified config unusable, it's a rare occurrance, and you're pointed to it by pacman with it's pacsave warning, which is of course basically superfluous. If it'd instead generate a pacnew warning, however, either because of the whitelist approach or because of lilo.conf being in the NoUpdate section, you'd have to look at the file to make sure it was _not_ a change that'd leave your modified, current file unusable. See the difference?

What do we learn? In either case, whitelist or blacklist, you'll need to have a look at the new file. If you forget or ignore that, you will get problems. The chance with the existing blacklist might be a little higher, but that really doesn't matter. If your system is important enough that you need a feature to help preventing the admin's ignorance, the system is also important enough to simply not be ignorant about pacman's warnings.

Additionally, if a file is listed as NoUpgrade, pacman will _always_ create a pacnew file, which is irritating. It should not generate a pacnew if the original file is the same as the new file to prevent a mixup with _really_ new files that'd need human attention. Bug? Feature? Who knows. At least this isn't documented, but can easily be reproduced; Just fiddle a bit with your lilo.conf, moving it, upgrading it, watching what happens. It'll make a lot of things clearer. Don't forget to restore it later, tho.

Greets,
Dennis

xirus · 2003-07-13 21:32:12

Ok, ic

But for me this would be unacceptable. This solves almost none of the issues I pointed out in my first post! If your lilo or grub file is modified and you MISS this fact, your system won't boot.

There are problems with both the way it currently works and what I suggested at first. That's why we should start thinking about a fbsd/debian-like system like andy suggested, this would solve all our problems because you HAVE to read about the differences in the files before you even CAN upgrade the program.

It should not generate a pacnew if the original file is the same as the new file to prevent a mixup with _really_ new files that'd need human attention. Bug? Feature?

I'm sure this can be made in a way that this problem is also solved.

I'm sorry I don't know any c, or otherwhise I'd really try to create something myself to help out the developers

Gyroplast · 2003-07-13 21:58:31

If your lilo or grub file is modified and you MISS this fact, your system won't boot.

True, that was your problem. This rises the question, though, why you would miss this fact if it's clearly that important? And to acerbate the issue, you still have the same problem if lilo would change it's config syntax and thus invalidate your current lilo.conf. Indeed, this is unlikely to happen, but the conclusion I can draw from this is that if you run a system that must not break after an update, you must not "miss" such crucial things as pacman's warnings. This simply must not happen, period. If it does, you shouldn't run a system being that important, strictly speaking. And if your system is unimportant enough to not pay pacman the due respect, you'll live with a non-booting system.

Naturally this is an uncomfortable situation; No man is perfect. Now the question is, should pacman really offer a more "into your face" approach? Or should it stay meek and only generate the little pacsave warning if it just fucked up your system? Should it maybe stop execution and ask whether it should overwrite? If interactivity is enforced, noone could miss such changes. Maybe a simple statement after the whole upgrade process, listing all pacsaved config files would already be enough. Write this statement or the whole pacman output to a file, so it does not scroll out of the way.

Whatever you do, you still need someone who cares. If you don't care what pacman is doing, you will mess up your config, no way around that. If you _do_ care, piping pacman's output into a log to make sure nothing is lost in the buffer, and a quick check for changes afterwards should be enough as most of the time you won't need to merge files anyway.

Am I missing something?

Greets,
Dennis

xirus · 2003-07-13 22:14:15

why you would miss this fact if it's clearly that important

I think I've given enough examples of how someone can miss this (yes, logging is a solution, but come on, who does this? )

Should it maybe stop execution and ask whether it should overwrite?

Yep, there should at least be an option to let it do this (which you can also turn of) - but as I said, it's even better to take care of it right away with a fbsd like system.

Yep, pacman works fine atm, but as I said, Arch is moving further and I think this is an area where is can improve a lot.

As I see it, everybody has another opinion about this (sarah32 and dariball prefer the old way, you want to loose the NoUpgrade option, I and andy seem to be interested in the fbsd system, Mork II and apeiro in my initial proposal - although they haven't responded on the new issues)), which means that the current system is not adequate (and yes, I've prolly misspelled that word )

Btw pacman is already a lot better than other systems like debian/gentoo/rpm distros, but that doesn't mean it shouldn't continue improving and leaving the competitors behind

hApy · 2003-07-13 23:05:02

Personally I like the current system alot, the freebsd option also sounds like a really good eventual evolution as well. But perhaps for the meantime the docs could be updated to include clear warnings about these workings of pacman so at least it is very clear to new users what the behaviour will be, and what to watch for. (for example don't -Syu without really paying attention) Also, adding the grub menu.lst to the default list of NoUpgrade's would be a good idea. Increased documentation in the setup docs would help admins avoid learning this the hard way -- setting up your pacman.conf when you install arch is a good idea.

Hapy.

scottro · 2003-07-14 04:58:28

I'm also a FreeBSD fan (my usual O/S actually) and do prefer something that won't touch my config files till I ask it to do so. Gentoo also has something similar (and has put in something similar to mergemaster) as far as important files in /etc

As for missing messages--especially with Arch's ease of updating things, which is one of the reasons I like it so much, how many folks type pacman -Syu hit y for go ahead and do it then go do something else?

Scott

Gyroplast · 2003-07-14 05:01:40

FreeBSDs mergemaster really only is a frontend for sdiff, no black magic, so it'd probably be well possible to port it to arch, especially as it's only a shell script.

I think I've given enough examples of how someone can miss this

Yeah, and I could easily construct a dozen more, but there's no excuse for not paying attention to the output of a crucial program if you cannot risk your system going haywire. If you _can_ risk it, though, your point is moot.

The inherent problem is that you cannot fully automate the process, thus always requiring manual interaction if a new config file differs from the original file and the current file has been modified. There is no way around that. You will shredder your system if you do not pay attention when this scenario happens, with the proposed change a little less often, but still. Since problems are not totally ruled out, you still have to check just as often as before, because you cannot know in advance how basic the config file changes are. If you still have to do the same amount of work, I don't see a reason to change anything.

I do not believe in "some security". Either pacman is able to do the job of upgrading reliably and automatically, or it is my sole duty to ensure configuration integrity. All pacman has to (and already can) do in that case is telling me that my interaction is required, and where. It does that now already (if I leave the NoUpgrade misbehaviour aside).

The mergemaster tool would be a very helpful _additional_ tool that could be offered, I have nothing against that, but it's not part of pacman. As I wrote it's just a frontend to sdiff, a fancy tool that may or may not be used. One job - one tool. Pacman's job is installing packages, not solving impossible dilemmas, therefore I'd vote against this extra functionality and would rather discuss how to improve administrator notification to reduce the chance of changes going unnoticed.

A solution to this underlying core problem would be "statistics" at the end of an upgrade process, stating if and what files have been replaced with newer versions. That's legible, it's easy to implement, and it does the job. If you're running pacman unattended, you're acting irresponsible. No matter if your config is replaced or not, you risk your system's integrity, the former risks it a bit more, the latter a bit less, but it's nevertheless risked, which is per se inacceptable, thus leaving the difference in reliability being merely a moot point.

I'll rephrase that:
Yes, the system will probably break more often if the current implementation is kept, but only if the administrator does not do his job and check pacman's warnings. There are no superfluous warnings being generated by pacman that would cause an admin to disregard the output, the only (documented) case of a pacsave being generated is when default config parameters have been changed in the package, which needs interaction _every time_. You're not winning anything by the change, you just gain a "safety net" that is not sufficient for the situation you are constantly referring to; A server system that simply must work. Therefore the safety measure is useless by concept. You cannot reliably help an admin who does not care, and if an admin does not care, it's by defintion not important enough to be considered, so why should a function be implemented that would solve (some, but not all) cases that are considered as unimportant by the admin anyway?

Whoa, evil topic here, so I say "Port mergemaster, make pacman a bit more verbose, be done with it." to keep it short.

Greets,
Dennis

Gyroplast · 2003-07-14 09:00:49

Hey, looks like we are totally forgetting another option!

In case of original, current and new file all being different, don't install the package at all. Print out a warning stating that this and that package could not be installed, and that these packages would need to be forced. That way one can guarantee that the admin has to be aware of the config file conflict.

Greets,
Dennis

xirus · 2003-07-14 09:13:17

That's also a way to go of course. An option to only update packages that don't use a modified config file. I suggest that IF we also do this (just as an optional thing) that this is the preferred method for newbies!

They can safely pacman -Syu and get the latest software (at least until a (modified) config file changes, but they could get new versions of that software if a new arch version comes out for example), without having to worry about packages not working, their system not booting, or messing around with some weird sdiff program.

But as I said: this should be optional, I still think that powerusers will benefit from a fbsd like solution, and if you think it's even easy to implement, then why not?

Mork II · 2003-07-14 10:29:45

There have been many good points in this thread. I will attempt a summary off the stuff I think is important.

1. Default pacman behaviour (if NoUpgrade is not used) when a new config file (Z) ,that differs from the original file (X), is available is to replace the current file Y with Z and move Y to pacsave. If X=Z but Y !=Z this does not occur. This causes problems when:
- The admin misses the pacsave warning. This is a problem since every system is important for the user/s no matter how short attention-span the admin has.
Suggested solutions:
a) "UpdateConfig: y/n" option. "Yes" giving current behaviour. "No" solves the problem of user configs beeing replaced but introduces the problem of stale config files (2) (if new parameter introduced in Z is lacking, program misbehaves).
b) "DoNotUpdatePackagesWithUpdatedConfigs: y/n". No giving current behaviour. Yes causing packages with changed config files to not be updated unless forced. Makes the admin pay attention. This means the packager must be aware of changes in the config files. Makes it harder to run pacman unatteded (wich is really nice when you have other stuff to do). ///Edit: To clarify. This only concerns packages that need an altered config file for running (X=!Z). All other packages still should be updated with a simple pacman -Syu. Did I get it right =X¥®µ§=?///
c) MergeMaster. Stop pacman to ask wether to install a new config file. Compare X, Y and Z? Pacman cannot be run unattended. Possibly confusing to use (I'm confused).
d) Verbose warnings. Simply print all pacsave warnings at the end of a pacman session and and point out in docs that these should be heeded.

2. If "NoUpgrade" is used pacnew files are generated. This would also happen with "UpdateConfig: No". Pacnew file is generated with every new update. No matter if X=Z.
Problems:
- Pacnew files are generated when they should not be.
- If the admin doesn't heed an important (x !=y) pacnew warning programs might misbehave.
Fixes:
a) Only generate pacnew when x !=y. More work for the packager.
b) All pacnew warnings showed at the end of a pacman session.

After giving it some thought I would prefer the following default behaviour: "UpdateConfig: No" (1. a) and fixed pacnew warnings (2 a). Gathering all warnings at the end of a pacman session would also help (2 b).

In any case (exept MergeMaster?) the admin must remeber wich files s/he has tampered with.

Xentac · 2003-07-14 16:11:36

I'd say (not that I know the pacman code) that it wouldn't be too difficult to add an interface (show diff, new, old, edit) to pacman's upgrade feature. Though, personally, I don't see that as being a problem, so I wouldn't want it changed. Maybe a config option to do one type of upgrade or the other. Call it InteractiveUpgrades = Yes/No or something...

Arch Linux

#1 2003-07-12 18:44:19

Request for a "NeverUpgrade" option for pacman.

#2 2003-07-12 19:46:52

Re: Request for a "NeverUpgrade" option for pacman.

#3 2003-07-12 20:43:14

Re: Request for a "NeverUpgrade" option for pacman.

#4 2003-07-12 20:45:16

Re: Request for a "NeverUpgrade" option for pacman.

#5 2003-07-12 21:35:12

Re: Request for a "NeverUpgrade" option for pacman.

#6 2003-07-13 11:40:18

Re: Request for a "NeverUpgrade" option for pacman.

#7 2003-07-13 11:59:30

Re: Request for a "NeverUpgrade" option for pacman.

#8 2003-07-13 16:49:57

Re: Request for a "NeverUpgrade" option for pacman.

#9 2003-07-13 20:09:43

Re: Request for a "NeverUpgrade" option for pacman.

#10 2003-07-13 20:55:14

Re: Request for a "NeverUpgrade" option for pacman.

#11 2003-07-13 21:32:12

Re: Request for a "NeverUpgrade" option for pacman.

#12 2003-07-13 21:58:31

Re: Request for a "NeverUpgrade" option for pacman.

#13 2003-07-13 22:14:15

Re: Request for a "NeverUpgrade" option for pacman.

#14 2003-07-13 23:05:02

Re: Request for a "NeverUpgrade" option for pacman.

#15 2003-07-14 04:58:28

Re: Request for a "NeverUpgrade" option for pacman.

#16 2003-07-14 05:01:40

Re: Request for a "NeverUpgrade" option for pacman.

#17 2003-07-14 09:00:49

Re: Request for a "NeverUpgrade" option for pacman.

#18 2003-07-14 09:13:17

Re: Request for a "NeverUpgrade" option for pacman.

#19 2003-07-14 10:29:45

Re: Request for a "NeverUpgrade" option for pacman.

#20 2003-07-14 16:11:36

Re: Request for a "NeverUpgrade" option for pacman.

Board footer