You are not logged in.

#1 2010-03-21 13:02:32

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

pacnew_scripts: scripts to merge some .pacnew files after an upgrade

info page: http://xyne.archlinux.ca/info/pacnew_scripts

I was updating my old system which doesn't get turned on that often and had to deal with about 15 pacnew files. Most only required a simple "mv /path/to/file{.pacnew,}" but some required merging. I found a script that I had written a while ago to merge /etc/group.pacnew, made some changes, made a copy which can handle gshadow, then wrote scripts to handle the pacman mirrorlist and locale.gen.

Thus the pacnew_scripts package was born.


infopage wrote:

This package contains scripts which can help merge .pacnew files after a pacman upgrade. These scripts print their results to STDOUT and thus do NOT make any changes to the files. It is up to the user to inspect the output and, if satisfied, redirect it to the original file and then delete the .pacnew file.

The scripts accept "-c" and "--compare" as options and will display relevant information about differences between the files. For group and gshadow, this will display 3 columns to compare the original, .pacnew and merged files and will use colors to highlight changes. For the mirrorlist and locale.gen, it will check that all settings in the original file are available in the new file and print out a list of any that aren't (e.g. if one of your current mirrors has been dropped from the mirrorlist, it will be listed in the output).

I'll probably add scripts as I go. I might also merge the scripts to avoid some redundancy but I decided to keep them simple for now. merge_pacnew_locale_gen is actually very generic and works on the mirrorlist and probably other files as well. If you find any, let me know.

Last edited by Xyne (2010-03-21 13:03:34)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#2 2010-03-21 13:50:37

Daenyth
Forum Fellow
From: Boston, MA
Registered: 2008-02-24
Posts: 1,244

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Doesn't pacman-contrib have some stuff for this?

Offline

#3 2010-03-21 15:16:45

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

I don't know and I don't really care. If it does and you find it more useful, use that instead. wink

This is yet another case of "I find this useful so I'll share it in case someone else does too".


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#4 2010-03-21 15:34:32

beroal
Member
From: Ukraine
Registered: 2009-06-07
Posts: 325
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

"diffpac" does this. It uses "vimdiff" for showing differences and merging. It also deals with *.pacsave files. But its interface is not perfect: redundant questions; some answers must be followed by "Enter" and some not.


we are not condemned to write ugly code

Offline

#5 2010-03-21 16:15:16

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

*sigh*

So use diffpac.


I really don't understand why people feel the need to post such comments. I actually find it surprising. Have you even tried these scripts and compared them to the alternatives that you've proposed? I myself find the group and gshadow scripts useful in particular, including the information provided by using the "--compare" flag.


Rather than be so eager to propose an alternative with the implication that this is pointless, either ignore this or try it yourself so that you can at least post comments based on an informed comparison.

Here's an analogy:

Someone shows up with a fresh batch of cookies and says "Hey, I just baked these with a new recipe that I came up with. I really like them. If anyone wants to try some, just help yourself". Someone walks by and without even looking at the cookies says "Hey, doesn't my grandmother already bake cookies?", then someone else shows up and says "Someone else already baked some cookies using the same ingredients that you've used."

If you don't want to try the cookies, then don't, but don't stand around implying that other cookies are better if you haven't tried them. Just move along and go get some of those cookies instead, and leave these to people who will enjoy them.


*edit*
This mentality is also one which impedes things in general. Too many people assume that simply because there is already something to do x, then there is no point in ever trying to create something which does it better.

Last edited by Xyne (2010-03-21 20:57:38)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#6 2010-03-21 20:37:29

Army
Member
Registered: 2007-12-07
Posts: 1,784

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Thanks Xyne for this package, I always merged these files by hand, didn't even consider searching for a tool doing this job. Now I even have two which I can chose from, which is great. Really, +1 for your criticism! There are plenty of tools out there which do similar jobs, I don't have to come up with examples. Right now we seem to have only ONE other tool doing your's job, so what's the big deal about it?

Offline

#7 2010-03-21 20:59:23

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Thanks, Army.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#8 2010-03-21 21:06:15

Arm-the-Homeless
Member
Registered: 2008-12-22
Posts: 273

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

There is a typo on the page for the tool. It says:

merge_pacnew_mirrorlost - for /etc/pacman.d/mirrorlist

Offline

#9 2010-03-21 22:57:41

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

updated, thanks


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#10 2010-03-22 00:44:22

beroal
Member
From: Ukraine
Registered: 2009-06-07
Posts: 325
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

I'm really astonished that you interpreted my post as an advertisement of "diffpac".

Xyne wrote:

Have you even tried these scripts and compared them to the alternatives that you've proposed? … you can at least post comments based on an informed comparison.

I expected such a comparison from you. If "pacnew_scripts" merges files automatically, what algorithm does it use?


we are not condemned to write ugly code

Offline

#11 2010-03-22 01:37:06

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

beroal wrote:

I'm really astonished that you interpreted my post as an advertisement of "diffpac".

Xyne wrote:

Have you even tried these scripts and compared them to the alternatives that you've proposed? … you can at least post comments based on an informed comparison.

I expected such a comparison from you. If "pacnew_scripts" merges files automatically, what algorithm does it use?

I didn't interpret it as advertisement. It seemed that you read the basic description of what this does and concluded that something else already does the same thing. Following on the tail of the previous posts I felt that your post was somewhat rude because it implies that this is useless. Obviously that is relative, but even so I see no point of posting in this thread to express it, even indirectly.

I admit that I may have interpreted your post the wrong way and if I did then I apologize, but even then I don't understand why you felt the need to post what you did in a contribution thread.


As for algorithms, there are different scripts and they use different algorithms. The group and gshadow scripts create hash representations of the groups in the original and .pacnew files. Groups which only exist in one of the files pass through untouched. Groups which exist in both have their members combined, which usually means that the original simply passes through. Using the "--compare" option, three aligned columns are displayed which clearly shows existing groups, new groups and the results of merging, with colors used to highlight all changes.

The mirrorlist script creates a list of selected servers in the current mirrorlist and generates a new mirrorlist from the .pacnew mirrorlist with all servers in the generated list uncommented, if they exist. Passing the "--compare" option will simply check if all the currently selected mirrors still exist in the new mirrorlist (they might not if they're dropped) and prints out a list of any that aren't.

The locale_gen script does the same thing for currently enabled locales.


Also note that all of the results are printed to STDOUT so no files are actually changes. It isn't fully "automatic" and the user is expected to sanity-check the results before overwriting the target files.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#12 2010-03-22 11:51:31

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Interesting concept. I use a simple bash loop that uses meld (graphical text merge tool), but since it's not aware of what the text means, everything must be done manually.  It seems your scripts can make smarter decisions because they "know" what the text means.
I will try this.

On a related note: I keep everything in my ~ in svn (this problem goes for every VCS) ; and frequently when I run a svn diff I see some configs have relevant differences (mostly the ones I did myself), but others are irrelevant. Ie some apps update last used geometry, last visited bookmark (yafc), etc. This is stuff that could be reverted instead of committed. I cannot just ignore the file because the same file holds relevant information as well. (actually I proposed to the XDG guys to keep state in a separate file, not much response. but that's another story)
I've had the idea for a long time to write a script that would do a svn diff on a file, and determine whether all changes are irrelevant or not.
I've never actually did it, but it seems along the lines what you're doing. Any input?


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#13 2010-03-22 12:51:51

dmz
Member
From: Sweden
Registered: 2008-08-27
Posts: 881
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

I totally agree with your third post. This ain't nothing I have no use for myself, but you write very sexy perl code and it's always nice reading it. Keep it up.

Offline

#14 2010-03-23 00:02:53

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Dieter@be wrote:

On a related note: I keep everything in my ~ in svn (this problem goes for every VCS) ; and frequently when I run a svn diff I see some configs have relevant differences (mostly the ones I did myself), but others are irrelevant. Ie some apps update last used geometry, last visited bookmark (yafc), etc. This is stuff that could be reverted instead of committed. I cannot just ignore the file because the same file holds relevant information as well. (actually I proposed to the XDG guys to keep state in a separate file, not much response. but that's another story)
I've had the idea for a long time to write a script that would do a svn diff on a file, and determine whether all changes are irrelevant or not.
I've never actually did it, but it seems along the lines what you're doing. Any input?

It obviously depends on the file format but I would try to create a representation of significant settings and then only compare those. It might be as simple as a few greps and a diff in bash or it might requires some internal representation and comparison (as I've done for the group file). If it's in XML then I would probably use Python's minidom to load the different versions then drop all child notes for settings that I consider irrelevant, although I might just read it as plaintext and use regexes to remove those nodes before a simple comparison. It really depends.

dmz wrote:

I totally agree with your third post. This ain't nothing I have no use for myself, but you write very sexy perl code and it's always nice reading it. Keep it up.

Thanks. I sometimes wonder what other Perl coders would think of my coding "style" (if you can call it that).


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#15 2010-03-24 17:49:41

pie86
Member
Registered: 2009-09-06
Posts: 78

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Great scripts!thanks!

Offline

#16 2010-03-24 18:14:38

Misfit138
Misfit Emeritus
From: USA
Registered: 2006-11-27
Posts: 4,189

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Great work, Xyne. Go, go go.
And thanks.

Offline

#17 2010-08-17 13:56:49

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Hmm, on my mirrorlist (see below), merge_pacnew_mirrorlist outputs the exact
same data as what's in /etc/pacman.d/mirrorlist.pacnew
ie, all entries are commented, even though i have 4 uncommented ones.
I was expecting the output to be the mirrorlist.pacnew, but the mirrorsthat I use uncommented. (and if they are not found - because i have i686 hardcoded instead of $arch, that they would just get added at the top or something)

My mirrorlist:

#
# Arch Linux repository mirrorlist
# Generated on 2009-12-06
#

Server = ftp://archlinux.mirror.kangaroot.net/pub/archlinux/$repo/os/i686
Server = http://archlinux.mirror.kangaroot.net/$repo/os/i686
Server = ftp://mirrors.kernel.org/archlinux/$repo/os/i686
Server = http://mirrors.kernel.org/archlinux/$repo/os/i686

And a thought:  your scripts could maybe use a -v flag,
which tells you what it does as the script is executing (reordering groups, sorting
usernames, importing new entries, ..).  doesn't seem very useful other then looking fancy, so i'm not even sure myself if it's a good idea.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#18 2010-08-17 20:09:43

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

I've updated merge_pacnew_mirrorlist to match architectures to $arch. Instead of uncommenting the matching mirror in the new file, it will replace it with the current hardcoded URL, i.e. it is left to the user to insert the $arch variable.

The output of "-c" also provides information about this now:

The following servers are missing from the new file:
  http://example.com/$repo/os/$arch

The following servers appear to match servers in the new file:
  ftp://archlinux.mirror.kangaroot.net/pub/archlinux/$repo/os/i686
    --> ftp://archlinux.mirror.kangaroot.net/pub/archlinux/$repo/os/$arch
  http://mirrors.kernel.org/archlinux/$repo/os/i686
    --> http://mirrors.kernel.org/archlinux/$repo/os/$arch
  http://archlinux.mirror.kangaroot.net/$repo/os/i686
    --> http://archlinux.mirror.kangaroot.net/$repo/os/$arch
  ftp://mirrors.kernel.org/archlinux/$repo/os/i686
    --> ftp://mirrors.kernel.org/archlinux/$repo/os/$arch

Those mirrors were not added to the top of the file because they did not appear to exist in the new file. The point of the script is to sync the mirrorlist to the new file only, so it removes old mirrors by design. I could even argue that mirrors with the $arch architecture should be removed, but obviously I see the utility of detecting them.

I don't think a "-v" flag would be useful. During normal operation the output is intended to be redirected so it would just clutter the output, even if I write the extra messages to STDERR. It might be useful when comparing files, but in that case I may as well just add the messages to the comparison itself, but I think the output of the comparison is already clear enough. For example, merge_pacnew_group clearly shows the changes using 3 columns and colored output when using "-c".

Any automation of the scripts should probably display the output of "-c" to the user and then prompt for confirmation before proceeding.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#19 2010-08-17 20:47:20

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Thanks, the improved matching is pretty much what I was expecting.  I get a good result now.

About mirrors listed in mirrorlist but not in mirrorlist.pacnew: what when the user uses a mirror that is not an official mirror? like a local mirror, or an unofficial mirror (proaudio, games, ..)?  It seems kinda dirty to configure  mirrors in the mirrorlist if they are officially recognized and in pacman.conf directly otherwise, especially if you want them to be in a very specific order.  But maybe i'm missing something.
What maybe would be a better approach, is for merge_pacnew_mirrorlist to have an interactive mode where you just say "unofficial mirror found.  keep it?" and the user hits y/n.

Too bad I just cleaned up all my files today, I hadn't tried the -c flags earlier so now I'm curious.  But from what you describe it looks it does the job.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#20 2010-08-17 21:36:18

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Dieter@be wrote:

About mirrors listed in mirrorlist but not in mirrorlist.pacnew: what when the user uses a mirror that is not an official mirror? like a local mirror, or an unofficial mirror (proaudio, games, ..)?  It seems kinda dirty to configure  mirrors in the mirrorlist if they are officially recognized and in pacman.conf directly otherwise, especially if you want them to be in a very specific order.  But maybe i'm missing something.
What maybe would be a better approach, is for merge_pacnew_mirrorlist to have an interactive mode where you just say "unofficial mirror found.  keep it?" and the user hits y/n.

The proaudio and games servers shouldn't be included in the official mirrorlist anyway. They're for completely different repos. wink (and if they actually do mirror [core] & co. then they should be added to the official list)

Custom mirrors (e.g. local) would indeed be removed, but if you have a custom mirror then it is unlikely that it would ever change so you may as well create your own custom mirrorlist and use that above the official one in pacman.conf. All you need is a separate file and a few extra "Include" lines.

As already mentioned, the user is supposed to check the output of the comparison then make whatever changes are deemed necessary before replacing the original file. That stated, I may consider adding an interactive mode (-i, --interactive).


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#21 2010-08-18 07:36:33

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Your custom mirror explanation is fair enough.  In fact, if a user configures his system according to this paradigm (local mirrors that have core etc in a separate mirrorlist file(*), and official mirrors in the real mirrorlist file), then there is no need for an interactive mode anymore.   Because your script can be sure it does the right thing.  Besides, I think running vimdiff <mirrorlist file> <pacnew output> is much more powerful then an interactive mode anyway.


(*) Putting something like the following in your pacman.conf seems to work as well:

[core]
Server = <local mirror here>
Include = /etc/pacman.d/mirrorlist

< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#22 2010-08-18 12:12:37

keenerd
Package Maintainer (PM)
Registered: 2007-02-22
Posts: 647
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

This is pretty neat.  Mind if I try incorporating it into pacmatic?  Right now pacmatic calls out to Pacdiff, but your scripts look very convenient.

Offline

#23 2010-08-18 12:32:37

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

keenerd wrote:

This is pretty neat.  Mind if I try incorporating it into pacmatic?  Right now pacmatic calls out to Pacdiff, but your scripts look very convenient.

you might want to incoporate arch-configs into pacmatic (https://bbs.archlinux.org/viewtopic.php?id=103028) instead.
arch-configs is a script that finds pacnew/pacsave files, and for each found file, runs pacnew_scripts where appropriate, and gives the user some options (mv merged file over original, run vimdiff, etc).  if no pacnew_scripts handler is found, you can do a manual vimdiff with the pacnew/pacsave file.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#24 2010-08-18 15:33:48

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

Dieter@be wrote:

(*) Putting something like the following in your pacman.conf seems to work as well:

[core]
Server = <local mirror here>
Include = /etc/pacman.d/mirrorlist

It does. What I meant by "All you need is a separate file and a few extra 'Include' lines." is this though:

[core]
Include = /path/to/local/mirrorlist
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /path/to/local/mirrorlist
Include = /etc/pacman.d/mirrorlist

[community]
Include = /path/to/local/mirrorlist
Include = /etc/pacman.d/mirrorlist

Of course, if you only have 1 local mirror then you may as well just insert it directly into the configuration file instead of creating a separate list for it.



@keenerd
I don't mind at all. Take a look at arch-configs too, as Dieter suggested.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#25 2010-09-25 16:47:49

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pacnew_scripts: scripts to merge some .pacnew files after an upgrade

I've added a script to merge /etc/shadow.pacnew.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB