Suggestion for new package management scheme

Mr Green · 2004-05-11 12:51:52

rlof

Mr Green....

cjdj · 2004-05-11 15:16:16

I'm fairly new to arch, and totally hooked. Fantastic. I've always liked linux, but never found a distro that I liked enough to actually use much. Arch fits the bill. Already converted 3 of my 4 systems to Arch. And the 4th is using Arch in a vmware virtual machine because it is a toshiba laptop and toshiba suck at supporting linux. Anyway...

So to add to the community (as a software developer), I'm creating some packages of the libraries and tools that I have written. Right now I am putting together a script that will help prepare an arch system for installing vmware. not a vmware package, but a vmware-prep package. Simple, not much to it, but a good start I figured for learning how to create a package.

So I've been thinking about what to do with them. Since currently they have limited use for others. I created my own repository on my website, and tested it, works quite well. And makepkg does seem to pull things from http even though it states that only ftp is supported.

pacman.conf:

[cjdj]
Server = http://www.cjdj.org/arch/pkg

Now I've also read in these forums that having a central repository is better than many user repositories, and I can understand the logic behind that. If package A depends on packages B and C and they are all in different user repositories, and repository B goes away (or pacman.conf not configured for it) then the chain is broken.

But at the same time, a single (containing current, extra, etc) repository with a bunch of mirrors seems to have some problems also. As it gets bigger the requirements for the mirrors become more restrictive. People can host a gig or two easily, but what happens when the repository becomes 20gig or more? (i have no idea what the current size it). The number of mirrors will get smaller. Not to mention the trouble it takes to co-ordinate and manage all the packages going into that repository. I would expect that the quality (or current-ness) of the packages would have to suffer as the number gets larger.

So what I was thinking... and the whole point of this post. Is in addition to a central core repository, we have a number of user repositories, that are referenced by the core repository. That way, the core repository has the majority of things that are most popular, and the user repos can have all the unique individual stuff (this has been mentioned before, a slim Current seems desirable). Now if someone does a pacman -Syu it would get the current db, and the extra, and then a list of other repos, and then check those. Right now, if you use the other repos, you have to update your pacman.conf file.

But with this method, you would still have the flexibility to use the current system the way it is, as well as spread the wealth to different repos for the not so popular stuff. If you dont want to check the extended repos, dont have that option set in the pacman.conf file.

So, in summary. We have a current repo. We have an extra repo. We have a config file stored somewhere on the current repo that has the rest of the user repo config data ([repo]nServer = ....), that is then checked against. In my mind I've been calling them User repos, but really they are extended repos, I guess.

Flames, comments all welcome, and sorry if this was already suggested but I missed it.

cjdj · 2004-05-11 20:35:45

well I thought about it some more, and figured that I could write something that can automatically update the /etc/pacman.conf file with the extended repositories that are listed in some controlled file.

So I spent the last hour or so on it, and am almost done. I have no idea if it will be useful or not, but I figured that instead of just saying what I think it should do, I would be better off doing it so people can see what I am talking about.

Also, I figured that the extended repository listing would be best kept as a package in itself so that it could be updated by pacman when it changes.

As it is, even if it is not adopted as a general thing, it could still be used by anyone who chooses to do so and it shouldnt break anything...

Anyway, as soon I can connect back up to my build system (darn VPN lost connection), I'll finish it, package it, and publish.

Ya Arch!

cjdj · 2004-05-11 21:44:09

ok, sorry for the 3rd post in a row here.

I've got my little project working, and I really do apologise if I am breaking any conventions or rules of etiquet here.

add this to your pacman.conf

[cjdj]
Server = http://www.cjdj.org/arch/pkg

and then

pacman -Sy
pacman -S pacmanex-data

It should try and install pacmanex and pacmanex-data.

and ... I'll post the PKGBUILDS in the group for those.

The source is somewhat ugly, but it is all in http://www.cjdj.org/arch/src/pacmanex/

Does anybody think this idea can help with the package situation?

Dusty · 2004-05-11 22:06:59

cjdj: Your posts are the first meaningful contributions this topic has seen.

I personally don't have need forthese tools, but people will find them useful and may add to them.

I also think that *unless* there are more developers welcomed to the team, non-central repositories for off-the-wall tools are a good fix; the user community can take care of itself.

Central repositories *are* better, but not necessarily in our current situatuon.

Dusty

cjdj · 2004-05-11 22:13:49

Thanks Dusty.

actually, something is wrong with my PKGBUILD's. even though the code has changed, it doesnt seem to be using it. There must be a cache somewhere I need to clear.

I'll be back to fix it in about an hour. Maybe people shouldnt try it yet until I get the repository working properly with the actual proper version... sorry.

cjdj · 2004-05-11 22:57:31

Ok, I dont know why it wouldnt use my new main.c file, it must be cached somewhere. So I renamed it. no biggie. pacmanex should be safe now.

So, here is what it does, and why I even bothered with this in the first place.

There is a lot of packages that people have built that are somewhat specific, and of very little use to the general public. But someone obviously found it useful to go to the hassle of creating the package. Over time these packages would take up considerable space in the 'extra' repository. At some point, the mirrors are not going to be able to host a huge repository.

So some of these specific packages should be put in a different repository. Which one? dosnt matter, create one. If you have access to a webserver, and make a package (that is not likely to be of interest to everybody), host it on the webserver.

Hopefully, this might solve some of the problems such as "what do I do with a package that I've made", and some of the packages made for the distro by publishers could then be hosted by the publisher reducing the amount of bandwidth needed by the archlinux mirrors (but would be much nicer of the publishers to actually mirror current and extra, but that might not happen)

Anyway, here is where pacmanex comes in. If you host your package on your webserver, everyone who uses arch would have to modify their pacman.conf file in order to be able to easily install your packages (and automatically update them). pacmanex will take a list of these extended repositories and automatically update pacman.conf with them. If a change to the extended list is made, pacman would automatically download a new list and update the pacman.conf file.

Some people might not like this... so, they have the option of simply not using pacmanex. But I do think that something like this should probably actually be a part of pacman.

So, if people like this idea, and want to use it, we can discuss how we control the list of repositories that pacmanex uses.

Thats about it. I think.

Dusty · 2004-05-11 23:11:23

cjdj wrote:

Anyway, here is where pacmanex comes in. If you host your package on your webserver, everyone who uses arch would have to modify their pacman.conf file in order to be able to easily install your packages (and automatically update them). pacmanex will take a list of these extended repositories and automatically update pacman.conf with them.

This idea and tool will definately have use, but including it in pacman is probably overkill. I understand that you would have the repository list updated to include all unofficial repos in pacman.conf? Two problems with this, users will be updating repo lists that they never use, taking time (Me --> dialup ), plus it may be unsecure, users won't know what repositories are available...

thanks for actually doing something though. I think bringing up the suggestion of more totally unofficial repositories might be worth something too, although I imagine some of the devs will be on my case about it...

Dusty

cjdj · 2004-05-14 16:31:31

Thanks Dusty,

Dusty wrote:

Two problems with this, users will be updating repo lists that they never use, taking time (Me --> dialup ), plus it may be unsecure, users won't know what repositories are available...

I guess I was thinking more about making some unofficial repositories a little more official without actually moving them into current, extra, etc.

For example, if vmware decide to create a package that can be used on arch, they can control their own repository.. for the products they want to provide, without having to go thru getting their packages into current or extra. As archlinux becomes more popular, hopefully more developers will start creating actual arch packages rather than arch volunteers and maintainers having to do this.

And since the config file that is merged into the pacman.conf file would be controlled in some manner, users having problems with a repository could easily report it to whoever is maintaining the extended repository list.

Anyway, its out there now.. and it seems to be working. If anyone wants to use it they can. You can download both pacmanex and pacmanex-data from http://cjdj.org/arch/pkg. All you need to do is install pacmanex-data and the post_install script will update your /etc/pacman.conf with the extra repositories (which is currently only my own little one).

markc · 2007-10-25 12:15:17

Excellent idea to make the extra repository info into a package itself. Including your pacmanex package then opens up the possibility of *automatically* including any number of other repositories and I'm all for package creation and deployment being distributed and as simple and hands-free as possible. The 2 main issues I see are of maintaining the list of extra repos and ensuring, somehow, that misc packages in the wild are not tainted with trojans or just plain badly hacked packages that can cause system damage when installed. A central interactive website could provide a method of adding/removing repo details that then gets converted into the pacmanex-data via a cron job on a regular basis. That kind of spoils the potentially nice distributed nature of your idea so perhaps a Git based system could be considered so at least every users installed pacmanex-data could be pushed/pulled to any other users Git repo... it *could* cut down on depending on a single central site to admin the pacmanex-data package. As for secure packages I can't think of anything really reliable short of some mythical signed package system tightly controlled by a central server by known and trusted developers. A rather militant and potentially un-open approach but we might have to suck on that one day as linux, in general, goes mainstream and attracts the attention of deviants.

Short of that, perhaps extra fields of meta data that determines the "level of trust" (LOT) of both the package and it's author could be added to package info (or at least the pacmanex extension somehow) and used a bit like how DNS and PGP servers are to domain and pgp key lookups. Some part of the web of trust needs to be centralised but it may only need to be simple UDP accessible lookups for a LOT number per package and author. If a package has a matching LOT above a certain threshhold then the end-users system could consider that package "safe enough" to install. A lower user selected LOT number threshhold means unstable/unsafer packages, a higher threshhold means the user will only accept more stable and safer packages. The LOT number could be determined by some form of voting or digg like system plus other factors like number of installs and reliability and trustworthiness of the author (which could be represented by another LOT number).

Perhaps using a bittorrent backend could be twisted to solve the package repo discovery and authenticity problems by providing the transport for the meta info as well as the packages. Anyway, I for one like the direction you are heading in and maybe a mashup of OpenID, PGP, Git and bittorrent for transport could provide a generic solution for any content, without ever requiring any sort of "official" DRM in the future. Using Arch packages as a prototype target to try and solve this problem space would be interesting.

Arch Linux

#76 2004-05-11 12:51:52

Re: Suggestion for new package management scheme

#77 2004-05-11 15:16:16

Re: Suggestion for new package management scheme

#78 2004-05-11 20:35:45

Re: Suggestion for new package management scheme

#79 2004-05-11 21:44:09

Re: Suggestion for new package management scheme

#80 2004-05-11 22:06:59

Re: Suggestion for new package management scheme

#81 2004-05-11 22:13:49

Re: Suggestion for new package management scheme

#82 2004-05-11 22:57:31

Re: Suggestion for new package management scheme

#83 2004-05-11 23:11:23

Re: Suggestion for new package management scheme

#84 2004-05-14 16:31:31

Re: Suggestion for new package management scheme

#85 2007-10-25 12:15:17

Re: Suggestion for new package management scheme

Board footer