[IDEA] Dealing with config file upgrades

jbNet · 2004-09-01 05:17:31

It is very annoying to have your config files thrashed every time you upgrade. Yes I realize your old ones are saved as backups, but what if there was an important change to the config file during the update? I have an idea on how Pacman could handle config file updates as elegantly as possible.

How it works:
1. Pacman knows pretty well what files are config files because of the "backup" option. Granted, not all of these are config files, but I think it's safe to use this field to decide which files to treat as config files. When one of these files is installed, a copy of the origional file is stored in the database (maybe even compressed).

2. When it comes time to upgrade the config file, pacman takes a diff between the origional config file and the current one, then attempts to apply the resulting .diff file to the new config file. Unless the upgraded config file is totally different, or the user has modified the config file massively, or it just so happens that the upgraded config file changes something near where the user did, the config file will automatically have changes merged into it painlessly. If there were any problems merging the new config file, a .rejected file could be dumped listing the problem areas for the user to manually merge, or ignore.

Example:
The sudo package has a config file /etc/sudoers. I (and many others) like to uncomment the line giving the users in the 'wheel' group access to sudo w/o a password. Say a new version of sudo is released w/ some extra samples down at the bottom of the config file, with the current system you would have to either move the .pacsave back into place, or modify the new config file. With this automatic system, you would have to do nothing, and would enjoy the benefit of the additoinal sample in your config file, but have your modified settings untouched.

This would also work in cases where there were no changes to the config file from one version to another. The diff would simply apply cleanly to the new file resulting in the exact same config file. This would also work in cases where the user doesn't modify the config, because the diff would simply be empty.

The potential problems I see are if the defaults in a config file are changed, which would cause those defaults to change in an existing config file, which could potentialy be a problem, and as I already said, in some circumstances the diff's wouldn't apply correctly requireing user intervention anyway.

Because of the first problem I listed, it would probably be a good idea to make this optional, maybe through an option in the pacman config file, and additionaly as a command line option. It would probably be best to have it on by default.

I would like to work on implenenting this if some people think it would be a good idea.

Any thoughts?

~Jake B

Bobonov · 2004-09-01 06:33:39

pacman has a complicated system to decide if it has to substitute a configuration file or not, it described in pacman man page.

HANDLING CONFIG FILES
pacman uses the same logic as rpm to determine action against files that are designated to be backed up. During an upgrade, it uses 3 md5 hashes for each backup file to determine the required action: one for the original file installed, one for the new file that's about to be installed, and one for the actual file existing on the filesystem. After comparing these 3 hashes, the follow scenarios can result:
original=X, current=X, new=X
All three files are the same, so we win either way. Install the new file.
original=X, current=X, new=Y
The current file is un-altered from the original but the new one is different. Since the user did not ever modify the file, and the new one may contain improvements/bugfixes, we install the new file.
original=X, current=Y, new=X
Both package versions contain the exact same file, but the one on the filesystem has been modified since. In this case, we leave the current file in place.
original=X, current=Y, new=Y
The new one is identical to the current one. Win win. Install the new file.
original=X, current=Y, new=Z
All three files are different. So we install the new file, but back up the old one to a .pacsave extension. This way the user can move the old configuration file back into place if he wishes.

The only problem with the diff method is when the conf file change from one version to another of the software.
For instance qwerty 1.9 has an option xyz in the conf file, in version 2.0 this option is deprecated or is changed in zyx, by doing a diff and a patch you break the configuration.
It is always better to import manually fancy options in the configuration.
If you do not want your conf file to be touched just put them in pacman.conf and afterwards apply the differences by hand.

bogomipz · 2004-09-01 09:53:51

Bobonov wrote:

The only problem with the diff method is when the conf file change from one version to another of the software.
For instance qwerty 1.9 has an option xyz in the conf file, in version 2.0 this option is deprecated or is changed in zyx, by doing a diff and a patch you break the configuration.

I don't quite follow you here. If an option is removed in the new version the patch will remove it in the config file, unless the user has already changed that line. The latter would make the patch not apply cleanly, so pacman would give up and tell the user to deal with it manually (go back to how it's done at present that is).

To me, this sounds like a very interesting idea. I'm not saying that it should be implemented right away, without carfully considering the consequences, but making pacman merge config changes when possible sounds like a smooth feature.

Bobonov · 2004-09-01 12:07:03

Here is an example:

(old.ori.conf)
workdir=/path/to/standard/dir/
standard_name="put here your name"
same_option="no change"
deprecated="old option"

(old.conf)
workdir=/path/to/my/dir/
standard_name="My name"
personal_option=true
another_personal_deprecated= "no valid option in new conf file"
same_option="no change"
deprecated="old option"

(new .conf)
workdir_new_option_name=/path/to/standard/dir/
standard_name="put here your name"
new_features="this is a new option"
same_option="no change"
substitute= "this substitute the deprecated option"

Try to make a diff from old.ori.conf to old.conf

1,2c1,4
< workdir=/path/to/standard/dir/
< standard_name="put here your name"
---
> workdir=/path/to/my/dir/
> standard_name="My name"
> personal_option=true
> another_personal_deprecated= "no valid option in new conf file"

And apply it to the new.conf, it give you an error.
Now try something that works.

(new .conf)
workdir=/path/to/standard/dir/
standard_name="put here your name"
new_features="this is a new option"
same_option="no change"
substitute= "this substitute the deprecated option"

Here is the result:

workdir=/path/to/my/dir/
standard_name="My name"
personal_option=true
another_personal_deprecated= "no valid option in new conf file"
new_features="this is a new option"
same_option="no change"
substitute= "this substitute the deprecated option"

That is a broken config

bogomipz · 2004-09-01 13:05:44

1) you should diff between old.ori.conf and new.conf so that no changes will give an empty diff (then try to patch old.conf)
2) when patch gives an error, you resort to how it's currently done
3) how is the last result broken? (it has "another_personal_deprecated", but that's because the user added something which's mentioned neither in the original nor the new config)

i3839 · 2004-09-01 13:06:25

I'm afraid Bobonov is right. Even if the patching went alright, you can't be sure that the new config makes sense and isn't broken. So you should manually check if the configs are alright anyway, throwing away any possible advantages, and instead now people get magically broken apps without them doing anything wrong. Automatic stuff should only happen when it's guaranteed that it works and won't break anything IMHO. It may fail, but then it shouldn't change anything, but here it can't know when it fails or not.

It would be better to handle all config files with some revision control system (e.g. CVS, Subversion or Arch). That should be smarter than just diffs and you can merge config changes much easier. It also provides a good backup system where you can restore all your configs to a certain date when they are known to work. If the changes are stored ("cvs commit") after each config change by Pacman, and perhaps each day with a cronjob, depending on how many backups you want, then that could give a lot possibilities for more advanced config handling. Slap an easy frontend to it so people can restore backups easily without knowing the internals of the revision system used. Only problem is that it's overkill. Personally I'm happy enough with the current config handling.

Safest and probably best way to go is to never touch user edited config files.

jbNet · 2004-09-01 17:42:05

Good point Bobonov, that may be a big problem. Thanks for the examples.

I'm trying to think how often that happens though, when a package changes the name of a config option? I can't think of any examples off-hand, but I'm sure it happens. Maybe this could still work, but be turned off by default. I think it would work best in the desktop situation, where not many config files are usually modified, and if one breaks, it's not going to open a security hole. I know I would much rather have my config files merged automatically 95% of the time, and have one break every once and awhile, rather than have to manually merge them every time I hit the X, Y, Z situation mentioned above. Now, in a server situation this is of course different, but Arch seems to be mainly used for the desktop.

If a package release was especially bad with this the upgrade scripts in the package could disable this functionality by moving the existing config file to config.old or something. Or maybe a warning message could be echoed during install of a particularly bad config file.

Also, the software using the config file usually does lots of checks on it when loading it. So if there was something that didn't make sence, there's a good chance the next time the software tried to read the config, it would give an appropriate error. And with depreciated options this is usually well documented, and done carefully, and gradually usually with a warning message from the software when parsing the config file.

In any case it would be useful for pacman to backup the origional config files in the database. Many-a-times when learning a new config file I manage to screw it up, and having a backup of the default config file to start over again would be very useful (you could of course just back up the file before hacking on it, but who does that? ).

And as far as using a revision control system for managing /etc/ This is a great idea for individual use, but probably should't be implemented as part of the distro. Plus afaik they probably use the same algorithms as diff and patch, so it would be essencially the same problem.

There are probably corner cases that could never be resolved automatically (at least w/ current technologies), but I believe this could still be useful for a desktop environment.

It would be pretty neat to be the distro with the best config file management.

~Jake B

jbNet · 2004-09-01 19:02:39

BTW those interested in this thread might also be interested in this project:

http://registry.sourceforge.net/

I know, I know, I had the same reaction to the name, but it really looks like a good idea.

Edit: looks like they actually changed their name

~Jake B

i3839 · 2004-09-01 20:51:40

I think their previous name was more fitting... The idea is exactly as good as MSWindows' registry.

jbNet · 2004-09-01 21:14:00

i3839 wrote:

I think their previous name was more fitting... The idea is exactly as good as MSWindows' registry.

Whoops, didn't mean to get off topic, I'll start a new thread on this:
http://bbs.archlinux.org/viewtopic.php?p=41581#41581

~Jake B

sarah31 · 2004-09-02 00:27:28

there is always the option for you to put certain files in the no upgrade section of pacman.conf :

# GENERAL OPTIONS
#
[options]
LogFile     = /var/log/pacman.log
NoUpgrade   = etc/passwd etc/group etc/shadow
NoUpgrade   = etc/fstab etc/raidtab etc/ld.so.conf
NoUpgrade   = etc/rc.conf etc/rc.local
NoUpgrade   = etc/modprobe.conf etc/modules.conf
NoUpgrade   = etc/lilo.conf boot/grub/menu.lst
HoldPkg     = pacman glibc

one of the problems is not being how to handle merging conf files but just which conf files to perform this on. i think this has been debated before so it may be time for a syncing of all the threads together and having a good debate about it.

tehdely · 2004-09-02 23:01:47

I think the most elegant system for handling this issue is currently Gentoo's "etc-update". If you're not familiar with it, portage automatically marks which config files have changed after an upgrade, and implores you to run etc-update.

I haven't used it myself, but the basic synopsis, as I understand, is that it presents diffs to you and gives you the option to merge them in some intelligent fashion, or to edit the file yourself.

Of course I think a feature like this would be better off as part of larger "post-install transaction" support in Pacman... something along the lines of packages pushing certain actions onto a global queue that must be reviewed by the user post-install.

pacman -S foobar foobar-plugins
Installing foobar... blah blah

You have three actions in the queue. Review? [Y/n] Y
1) update /etc/foobar.conf
2) configure foobar plugins
3) enable foobar support in apache

Let your imagination go with this one

sarah31 · 2004-09-02 23:32:39

the two weeks i used gentoo etc-update never worked and usually royally foobared things. I would hope that it is considerable better now. in fact it was NOT recommended to use it.

jbNet · 2004-09-02 23:33:04

I think the most elegant system for handling this issue is currently Gentoo's "etc-update"

I agree, I run a Gentoo server at work, and etc-update comes in handy when apache/samba/etc get updated. However it is still very annoying, and can be difficult to use until you get used to it. I like your idea about a post-install queue... I hate it when packages spit out important information after they are installed/upgraded to the console, but you miss it because it scrolled up too far (this is a bigger problem in Gentoo). Having the post-install queue would solve that problem in a great way. It would also make the messages persistant so that if you have pacman run in a cron job you could run a command like 'postinstq' in the morning and see if there is anything you need to do... or even something could be mailed to you (but that adds a transport as a dep to pacman, probably not a good idea).

Still, I would like to do even better than etc-update... I don't need to reiterate what I've said above. Maybe instead of enableing or disableing this feature, the option could be interactive mergeing or automatic merging. Where interactive would be simmilar to the etc-update, and automatic would be as described above.

~Jake B

Xentac · 2004-09-03 00:25:22

tehdely wrote:

I think the most elegant system for handling this issue is currently Gentoo's "etc-update". If you're not familiar with it, portage automatically marks which config files have changed after an upgrade, and implores you to run etc-update.
I haven't used it myself, but the basic synopsis, as I understand, is that it presents diffs to you and gives you the option to merge them in some intelligent fashion, or to edit the file yourself.
Of course I think a feature like this would be better off as part of larger "post-install transaction" support in Pacman... something along the lines of packages pushing certain actions onto a global queue that must be reviewed by the user post-install.
pacman -S foobar foobar-plugins
Installing foobar... blah blah
You have three actions in the queue. Review? [Y/n] Y
1) update /etc/foobar.conf
2) configure foobar plugins
3) enable foobar support in apache
Let your imagination go with this one

http://bbs.archlinux.org/viewtopic.php?p=33282#33282

Check out my bash scripting in there...

tehdely · 2004-09-03 01:39:17

jbNet wrote:

It would also make the messages persistant so that if you have pacman run in a cron job you could run a command like 'postinstq' in the morning and see if there is anything you need to do... or even something could be mailed to you (but that adds a transport as a dep to pacman, probably not a good idea).

I thought maybe it should use the same notification system that is used to remind you of when you have new mail when you log in ( how exactly is that done, btw ), but 'postinstq' sounds a lot more sensible. :oops:

Also, think of the idea of doing user-specific install actions as well. For example, applications which must be configured before first-run (on a user-wide level, not system-wide). This would be ridiculously useful for some of my GNUstep packages, especially given that a large deal of manual configuration is needed to get a usable "desktop" ( and I'm tired of walking people through it on irc )

jbNet · 2004-09-03 02:20:09

tehdely wrote:

I thought maybe it should use the same notification system that is used to remind you of when you have new mail when you log in ( how exactly is that done, btw ), but 'postinstq' sounds a lot more sensible. :oops:

Hrm, I think that maybe that is built in to bash? If not then it is in /etc/profile or something (but I can't find it in there). Anyway you could that that functionality by putting it in /etc/profile or ~/.bash_profile or etc etc. But it would have to be really fast or I know I wouldn't want it in there.

~Jake B

i3839 · 2004-09-03 12:42:33

I thought it was in /etc/issue, not sure though.

etc-update is a disaster, at least that's my experience.

jbNet · 2004-09-03 16:24:01

i3839 wrote:

I thought it was in /etc/issue, not sure though.
etc-update is a disaster, at least that's my experience.

actually /etc/issue is what shows up before you log in, no?

and at first I totally didn't get etc-update... but it works fairly well after you use it for awhile. It's better than trying to manually merge the files, but it isn't pretty.

~Jake B

butters · 2005-09-01 11:45:27

I'm going to awake this thread from the dead... I haven't ever used Archlinux (I'm a super-interested Gentoo user), but here's what I know about this stuff:

1) Pacman's config file handling is broken. From the manpage:

original=X, current=X, new=X
    All three files are the same, so we win either way. Install the new file. 
original=X, current=X, new=Y
    The current file is un-altered from the original but the new one is different. Since the user did not ever modify the file, and the new one may contain improvements/bugfixes, we install the new file. 
original=X, current=Y, new=X
    Both package versions contain the exact same file, but the one on the filesystem has been modified since. In this case, we leave the current file in place. 
original=X, current=Y, new=Y
    The new one is identical to the current one. Win win. Install the new file. 
original=X, current=Y, new=Z
    All three files are different. So we install the new file, but back up the old one to a .pacsave extension. This way the user can move the old configuration file back into place if he wishes.

XXX and XYX will never happen. Why? Because default config file always change from release to release, if only their version tags at the top. XYY will never happen because the user would never change the version tag on his/her own, besides this being extremely improbable in the first place. XYZ becomes a big problem, because X and Z could differ in a meaningless way (version tag, comments) and cause the modified config file to be saved in favor of the new file.

So we really only see XXY, which has an obvious correct response, and XYZ, where we never have a perfect response and sometimes for a silly reason.

Solution: sed s/#.*$//g
Remove all comments from the config file and then take the checksum. This will give much more meaningful file matching.

2) etc-update works. There's nothing about it that doesn't work as advertised. But if you mean to say that it is a pain to use, then you're pretty much right. That's why the Gentoo community created dispatch-conf, which solves everything mentioned in this thread. It automatically updates all configs that have never been modified or differ in CVS cruft or whitespace (or comments if that option is enabled), provides interactive merge and in-place editing features, and stores replaced versions in the form of either patches against current or through its own RCS tree.

3) Arch should offer two config file handling modes: one which is like the current system but with my suggested enhancement, and another which notifies the user of the presence of .pacsave files after any pacman operation and suggests running dispatch-conf. For instance, in /etc/pacman.conf: advanced_config_mode = [yes/no].

dtw · 2005-09-01 12:12:31

butters, i dunno that much about it but you really should find the bug on the bugtracker and suggest your ideas - there may be good reasons why they are no good but they look pretty sane to me

If you also ported dispatch-conf to work with Arch yourself then that would certainly help

phrakture · 2005-09-01 15:37:51

butters wrote:

XXX and XYX will never happen. Why? Because default config file always change from release to release, if only their version tags at the top. XYY will never happen because the user would never change the version tag on his/her own, besides this being extremely improbable in the first place. XYZ becomes a big problem, because X and Z could differ in a meaningless way (version tag, comments) and cause the modified config file to be saved in favor of the new file.

This all assumes the config file *has* a version tag. Take a look at /etc/vimrc or the conf file from the "slim" package.

3) Arch should offer two config file handling modes: one which is like the current system but with my suggested enhancement, and another which notifies the user of the presence of .pacsave files after any pacman operation and suggests running dispatch-conf. For instance, in /etc/pacman.conf: advanced_config_mode = [yes/no].

The fact of the matter here is that the devs don't want to do this and don't care that much... if *you* would like it, go ahead and script it - I'm sure alot of old gentoo users would love to have something like that under arch. But, my point is that, this is not something which needs to affect pacman/makepkg or anything - it can be entirely external.

butters · 2005-09-02 01:19:45

If config files don't have version tags, then XXX and XYX are much more likely and XYY is at least within the realm of possibility. However, these are not the problem cases. In XXX and XYY you can switch to the new file without worrying, and in XYX the user definitely wants to keep his/her modified config (although there is a small chance that some modifications are invalid for the new release).

The problem is that pacman is way too sensitive to meaningless changes between original and new config files, so it mistakes the XYX case for the XYZ case. In the former case the modified config is kept in place, while in the latter case it is replaced by the new config. If the checksum is taken on a temporary copy of the config files that have been stripped of comments and extraneous whitespace, then pacman can make a more accurate distinction between XYX and XYZ and do the right thing in more cases.

Unfortunately pacman is written in C, and the regex APIs for C are horrible.

phrakture · 2005-09-02 07:34:31

butters wrote:

The problem is that pacman is way too sensitive to meaningless changes between original and new config files, so it mistakes the XYX case for the XYZ case. In the former case the modified config is kept in place, while in the latter case it is replaced by the new config. If the checksum is taken on a temporary copy of the config files that have been stripped of comments and extraneous whitespace, then pacman can make a more accurate distinction between XYX and XYZ and do the right thing in more cases.

This is the same logic RPMs use... and if I recall, it's done the same way.

Really, it's not a big deal - I mean, I've never had a showstopper, except when I retartedly forgot to fixup my sshd config :oops:

Arch Linux

#1 2004-09-01 05:17:31

[IDEA] Dealing with config file upgrades

#2 2004-09-01 06:33:39

Re: [IDEA] Dealing with config file upgrades

#3 2004-09-01 09:53:51

Re: [IDEA] Dealing with config file upgrades

#4 2004-09-01 12:07:03

Re: [IDEA] Dealing with config file upgrades

#5 2004-09-01 13:05:44

Re: [IDEA] Dealing with config file upgrades

#6 2004-09-01 13:06:25

Re: [IDEA] Dealing with config file upgrades

#7 2004-09-01 17:42:05

Re: [IDEA] Dealing with config file upgrades

#8 2004-09-01 19:02:39

Re: [IDEA] Dealing with config file upgrades

#9 2004-09-01 20:51:40

Re: [IDEA] Dealing with config file upgrades

#10 2004-09-01 21:14:00

Re: [IDEA] Dealing with config file upgrades

#11 2004-09-02 00:27:28

Re: [IDEA] Dealing with config file upgrades

#12 2004-09-02 23:01:47

Re: [IDEA] Dealing with config file upgrades

#13 2004-09-02 23:32:39

Re: [IDEA] Dealing with config file upgrades

#14 2004-09-02 23:33:04

Re: [IDEA] Dealing with config file upgrades

#15 2004-09-03 00:25:22

Re: [IDEA] Dealing with config file upgrades

#16 2004-09-03 01:39:17

Re: [IDEA] Dealing with config file upgrades

#17 2004-09-03 02:20:09

Re: [IDEA] Dealing with config file upgrades

#18 2004-09-03 12:42:33

Re: [IDEA] Dealing with config file upgrades

#19 2004-09-03 16:24:01

Re: [IDEA] Dealing with config file upgrades

#20 2005-09-01 11:45:27

Re: [IDEA] Dealing with config file upgrades

#21 2005-09-01 12:12:31

Re: [IDEA] Dealing with config file upgrades

#22 2005-09-01 15:37:51

Re: [IDEA] Dealing with config file upgrades

#23 2005-09-02 01:19:45

Re: [IDEA] Dealing with config file upgrades

#24 2005-09-02 07:34:31

Re: [IDEA] Dealing with config file upgrades

Board footer