how to manage "forks" of official packages?

ataraxia · 2009-09-17 02:23:39

I'm trying to figure out a way to use a custom repository to track packages I copy from ABS and modify. Creating the packages, and the repo DB, is easy. I just can't figure out how to sanely USE such a repository.

What I'd like to emulate is a "highest version wins regardless of source repo" behavior, and I know pacman doesn't do this. I want name collisions of packages to resolve to my repo when myversion <= officialversion, and use the official package otherwise.

The value of this behavior would be that pacman alerts me (by nominating for upgrade) when the official package is updated and my modified version is thus out of date. Then, I cancel the pacman upgrade and go update my personal repo instead.

If I put my repo first in pacman.conf, I can't track the official versions of the modified packages. If I put it last, my repo adds no value over not having a personal repo at all and just doing "pacman -U" on the individual packages (which is what I'm doing now).

I want to make it clear that I'm NOT complaining about pacman's behavior, or asking for a change. I think rather that I'm going about this the wrong way, and so I'd like to know how others have solved this problem.

(This thread is also related to my "which official packages do you modify?" thread, in which I try to determine how important this problem is to other users.)

Xyne · 2009-09-17 02:36:13

Instead of messing around with repo hacks, why not just write a script that checks each package in your repo against the other repos and prints out a list of packages that are no longer up-to-date.
*edit*
If you keep the versions in sync with the official packages then you can use a simple comparison for the versions. If not, you will need to use vercmp as mentioned by Shining below.

Last edited by Xyne (2009-09-17 09:07:11)

shining · 2009-09-17 08:18:33

Xyne wrote:

Instead of messing around with repo hacks, why not just write a script that checks each package in your repo against the other repos and prints out a list of packages that are no longer up-to-date. The only complication is that the version comparison function is not a simple "<=". If you normally keep the versions synchronized then you could use a simple comparison. Otherwise you can take a look at bash-xyne-common_functions for a bash version and perl-xyne-arch's Pacman.pm for a Perl version.

For this use case, I think a simple != on the version strings is enough.

Otherwise, do you know about 'vercmp' ?

Xyne · 2009-09-17 08:37:38

shining wrote:

Otherwise, do you know about 'vercmp' ?

Well, I do now.

I should change my sig to "I've reinvented the wheel so many times that I'm getting quite good at it.".

*edit*
updated bash-xyne-common_functions to use vercmp instead of my bash implementation

Last edited by Xyne (2009-09-17 09:08:20)

shining · 2009-09-17 09:13:55

Xyne wrote:

shining wrote:
Otherwise, do you know about 'vercmp' ?
Well, I do now.
I should change my sig to "I've reinvented the wheel so many times that I'm getting quite good at it.".

ahah

There is one case where I did not want to call vercmp, because it had to be invoked many many times, and this was a performance problem. So I even wrote a small python bindings to be able to call it natively :
http://projects.archlinux.org/?p=dbscri … .c;hb=HEAD

otherwise, just for fun, you might want to test your own implementation with the test suite :
http://projects.archlinux.org/?p=pacman … sh;hb=HEAD

Since this is completely off-topic (sorry), we might want to take this offline..

Back on topic, I agree with Xyne, the only solution is see is to write a small script doing that. It should not be difficult.

comm -23 <(pacman -Sl custom_repo| cut -d' ' -f2- | sort) <(pacman -Sl core extra community | cut -d' ' -f2- | sort)
?

ataraxia · 2009-09-21 00:34:30

So, it's taken me a while to write a reply to this thread. I had the problem that when I saw the suggestions, I immediately didn't like them. It's taken me 3 days to figure out why not, and I felt that I'd better have a good reason before I reject the advice of both a pacman developer and a repo maintainer on this subject I do appreciate the replies, and I don't want to look like a jerk.

My feeling is basically that these answers don't make any improvement over what I've been doing - just not having my custom packages in a repo at all, but installing them with pacman -U. Adding shell scripts to my system is not very KISS. I'm a sysadmin by trade, and if there's one thing I've learned, it's to use as few tools as possible. This is why I avoid quality software like rebase, reflector, powerpill, and yaourt - great stuff, but I can get by without them. (Naturally, Xyne, being the author of several of those examples, probably doesn't subscribe to this philosophy.)

I wonder if I'm suffering from the XY problem here. I don't really want to use a personal repo to solve this problem (in fact I'd prefer to avoid it for the complexity it represents) - I just want to be able to fork official PKGBUILDs in a way that's lightweight, flexible, and scalable, and I saw a personal repo as a good shot at success. Perhaps readers of this thread could tell me how they handle this problem? I'm basically looking for methodology, not helper-tools.

Xyne · 2009-09-21 02:25:19

ataraxia wrote:

So, it's taken me a while to write a reply to this thread. I had the problem that when I saw the suggestions, I immediately didn't like them. It's taken me 3 days to figure out why not, and I felt that I'd better have a good reason before I reject the advice of both a pacman developer and a repo maintainer on this subject I do appreciate the replies, and I don't want to look like a jerk.
My feeling is basically that these answers don't make any improvement over what I've been doing - just not having my custom packages in a repo at all, but installing them with pacman -U. Adding shell scripts to my system is not very KISS. I'm a sysadmin by trade, and if there's one thing I've learned, it's to use as few tools as possible. This is why I avoid quality software like rebase, reflector, powerpill, and yaourt - great stuff, but I can get by without them. (Naturally, Xyne, being the author of several of those examples, probably doesn't subscribe to this philosophy.)

What a jer...

ataraxia wrote:

I wonder if I'm suffering from the XY problem here. I don't really want to use a personal repo to solve this problem (in fact I'd prefer to avoid it for the complexity it represents) - I just want to be able to fork official PKGBUILDs in a way that's lightweight, flexible, and scalable, and I saw a personal repo as a good shot at success. Perhaps readers of this thread could tell me how they handle this problem? I'm basically looking for methodology, not helper-tools.

My understanding is that you want to maintain custom versions of repo packages in the same release cycle. You also want to achieve this without using simple tools (or, by extension of your KISS argument, complex tools) and in the simplest way possible. I see that as a contradiction.

Even considering the XY problem, your original post suggested a willingness to accept a fair amount of complexity so it seems strange that you would reject a very simple script to achieve your goal. If doing it all manually is undesirable but using a script is unacceptable then I don't see a solution.

There's an unavoidable (if small) level of complexity involved in what you want to do. Unless you have figured out a way to automate building the custom packages with each new release with whatever changes you've made (which would certainly require complex tools), you will still need to do that manually. If you only need to install these packages on your own system then a simple "makepkg -irs" will work. If you need to share the packages then you have to "makepkg" and then <create a repo and host it|write scripts|use pkgd|do it manually|etc>.

The former is as simple as it gets. The latter will require tools or time. I opt for using tools in that case. In either case, the remaining issue is notification of the update cycle. To achieve that, you either have to check for newer versions manually or use a script. Your original post involved essentially hijacking pacman's update function using repo trickery in order to manually abort a download which you would later have to resume just to find out about new releases. That involves repo maintenance (creation, updating, cleaning) and at least 2 calls to pacman. A script to print out a list of available updates seems very much like an improvement to me. You've already stated that you don't see that as an improvement.

I suspect that I still don't fully understand what you actually want to achieve.

Last edited by Xyne (2009-09-21 02:29:37)

shining · 2009-09-21 09:18:31

I am also confused.
If you want to use as little unofficial scripts as possible, then keeping using the same way and using a one-line script to track down version changes between official repos and your repo seems the most appropriate.

If you want a more automated / complex in the code / simpler to use way, you could have a look at :
1) srcpac
http://www.archlinux.org/news/75/
http://projects.archlinux.org/?p=srcpac.git;a=summary
2) customizepkg
http://bbs.archlinux.org/viewtopic.php?id=53280
http://aur.archlinux.org/packages.php?ID=10314

ataraxia · 2009-09-21 14:06:26

I hadn't looked much at srcpac before. I didn't know about /etc/srcpac.d and its sed scripts. Those explain: a - why srcpac is useful, and b - the model upon which customizepkg is built. These are interesting, but I see a bit of limitation for bigger modifications - it would be a bit hacky to insert lines into the build function of a PKGBUILD to copy in additional patches, and that would basically be needed given that srcpac can't natively add source files to a build. (I wouldn't think too hard about this paragraph - see the next one.)

I think that the reason we're all so confused here is that I'm trying to do the impossible. In particular, I think I've already reached the global minimum of complexity for my given set of goals, and the going any further would be reducing the irreducible. I made this thread believing that I had only, so to speak, reached a local minimum, and somebody would present me with a totally different way of working than I had seen before. Unfortunately, it seems that I understood the problem space better than I had hoped. (Isn't it strange when you can hope you understand something less than you do?) I'm the sort of person who would reduce all my computing needs down to one simple case that handles everything identically if I could - sort of like one of those fraudulent compression algorithms we see every now and then which compresses everything down to just one bit if you run your data through it enough times.

I'd say that unless somebody has a totally unexpected direction for this to go in, there's no need to expend more effort on this thread. (And can you tell by the metaphors in this post that I've spent the last week reading Stephen Wolfram? )

Arch Linux

#1 2009-09-17 02:23:39

how to manage "forks" of official packages?

#2 2009-09-17 02:36:13

Re: how to manage "forks" of official packages?

#3 2009-09-17 08:18:33

Re: how to manage "forks" of official packages?

#4 2009-09-17 08:37:38

Re: how to manage "forks" of official packages?

#5 2009-09-17 09:13:55

Re: how to manage "forks" of official packages?

#6 2009-09-21 00:34:30

Re: how to manage "forks" of official packages?

#7 2009-09-21 02:25:19

Re: how to manage "forks" of official packages?

#8 2009-09-21 09:18:31

Re: how to manage "forks" of official packages?

#9 2009-09-21 14:06:26

Re: how to manage "forks" of official packages?

Board footer