Suggestions for INCOMING

contrasutra · 2004-04-22 19:07:22

Here's the thing. I really don't like the INCOMING system as it stands. The idea is wonderful, but it seems like it was haphazardly implemented.

1. I don't like having to download big packages to test them. I can't use the package anyway (dangerous and I have to test rebuilding), so it's wasted time/bandwidth. I think we should convert the system to a PKGBUILD-only system. People can tar up the directory that has the PKGBUILD, .install, patches, etc.

This will obviously help bandwidth/space. Also, it lets us see problematic packages easily by size. All too often, people are including loads of extra things in the tarballs, like the entire src/ and pkg/ directories.

Since INCOMING has really become just a holding ground for package testers, I would appreciate this accommodation.

2. The system is horribly disorganized. I've looked through and tested most of the packages I can (can't do things like drivers that I don't use, etc). There is simply more junk than useful things. This is due to duplicate packages, and some that are not made properly at all. I pretty much make the PKGBUILDs from scratch at this point.Here's my idea for how to fix the mess:

Make a dir in INCOMING for each TU. Then a contributor can upload the PKGBUILDs to the TU he wants. This will do a few things:

(a) It gives each TU a specific set of packages. You can easily gauge performance. TUs don't have to go hunting around INCOMING for something they can test, they only have to look at their folder (and can even automatically download it/test it, etc). Most of my testing time for INCOMING consists of finding a package that I can test, and that's not a dupe or not in a repo already.

(b) It keeps the testing distributed. A contributor wouldn't upload to a TU folder that is stuffed. He'd pick a folder that has less packages in it. You could also have a max file limit set, so a TU couldn't get too over worked.

(c) It distributed the cleaning responsibility as well. Give the TU write permissions for his folder, and he can keep things clean, so the Package Maintainers don't have to.

I know people have suggested lots of neat automated systems, but sometimes it's just better to throw a little manpower/organization at the situation. KISS.

I'd like your suggestions!

P.S. I'd also appreciate it if the devs could check STAGING a little more. It's getting very large.

contrasutra · 2004-04-22 19:14:35

Sorry to double post, but:

Xentac and I would both like better package building guidelines. I don't want anything as strict as Debians, but I feel there are some more things that have to be clearly defined. If anyone else agrees, I would be happy to write up a draft of the changes I would like.

Dusty · 2004-04-22 20:20:12

I'm all for it! Packaging guidelines can be strict, as long as they are really just guidelines. rules are made so you think twice before breaking them.

If you implemented this system, it might also be neat to add some sort of profile for each TU, so they can specify the sorts of packages they are most interested in (or capable of) maintaining.

Dusty

Egil.B · 2004-04-22 21:26:54

Sounds like a good idea to me.

Speaking of STAGING, sylpheed-gtk2 needs the libssl0.7 rebuild

contrasutra · 2004-04-22 23:42:34

If you implemented this system, it might also be neat to add some sort of profile for each TU, so they can specify the sorts of packages they are most interested in (or capable of) maintaining.

That's a great idea. Not that each TU would only be able to look at certain types of packages (like the Devs), but they could state their machine and what they're most experienced with.

Speaking of STAGING, sylpheed-gtk2 needs the libssl0.7 rebuild

Done.

LavaPunk · 2004-04-22 23:42:53

I like the idea, but am unsure how much I like letting users specify which TUR they'd like to give it too though. Just seems odd to me.

rasat · 2004-04-23 04:09:33

contrasutra wrote:

Make a dir in INCOMING for each TU. Then a contributor can upload the PKGBUILDs to the TU he wants.

To get the incoming more organized I fully agree. From AIR I can say its worthy to put some effort in this. There are wonderfully softwares in incoming. If a pkg is poorly presented / build (uploaded) getting disregarded by TUs, we may miss having one more good package.

To make an incoming dir for each TU will not be proper when most contributors don't know the TUs..... don't know whom to upload... in long run all packages will go to contrasutra

The matter is for TUs to know what to test among the 300 incoming. I suggest to use AIR... its a register (not a downloader) to know what are available and TUs can check what's new --> select type of pkg: "Incoming", select "View new pkgs", and press "Submit Search". The description, category and type of application will help TUs to choose instead of going through the incoming-ftp or "New package" forum.
http://bliss-solutions.org/archlinux/incoming/

PS.
Daily I am running one script checking what are new and recognizes the changes between AIR data and ftp repos (incoming and TUR).

rasat · 2004-04-23 14:19:45

contrasutra wrote:

I think we should convert the system to a PKGGBUILD-only system. People can tar up the directory that has the PKGBUILD, .install, patches, etc.

Addition to what I said, AIR can replace the incoming-ftp completely (if we want) as long as the contributors post the PKGBUILD in "New package" forum. In AIR all package names are linked to this forum (if available).

Example (click "PKGBUILD at arch forum"):
http://bliss-solutions.org/archlinux/in … .php?id=57

LB06 · 2004-04-23 18:39:19

What exactly are the differences between incoming, unstable and testing (or even staging) anyway?

From highly experimental to stable:
- incoming
- testing
- unstable
- TUR
- extra
- current
- release

Am I correct?

I find this large number of repository's very confusing. Making incoming PKGBUILD only could be a great thing to do to clear everything up. And besides what, what is the point of unstable now we have testing?

Dusty · 2004-04-23 19:02:38

LB06 wrote:

What exactly are the differences between incoming, unstable and testing (or even staging) anyway?
From highly experimental to stable:
- incoming
- testing
- unstable
- TUR
- extra
- current
- release
Am I correct?

Not quite a good outline. Incoming packages might be rock stable, but they are defined by never having been tested by anybody but the implementer. You have to be pretty brave (or dumb) to try such a package without checking the PKGBUILD yourself. This is why there isn't much point in putting anything but the PKGBUILD and possibly shellscripts not available anywhere in incoming, nobody should install the package from incoming anyway, only via ABS. Doing this would save space and download time.

TUR/staging is the next step, its tested by trusted people (trusted being defined by the trusted users themselves, hehe). These are stull unoffical packages.

testing, unstable, current, release, and extra, are official repos.

the difference between testing and unstable is that testing is for untested packages, while unstable is for tested packages that are developmental/broken in some way.

Current and release are exactly the same, except that current is more up to date. Release really doesn't count for anything (ie: ignore it) except that it represents a particular release on ISO. It shouldn't technically be any less or more stable than current, since it was current at one time.

Extra is for packages that are too big or uncommon to go into a normal release.

My understanding anyway. Each repo has a separate well-defined use. The only ones I'm not sure of a difference are staging/testing, the real difference is that testing is more official, but TURs are gaining officiality now anyway...

contrasutra · 2004-04-23 19:34:24

rasat wrote:

contrasutra wrote:
Make a dir in INCOMING for each TU. Then a contributor can upload the PKGBUILDs to the TU he wants.
To get the incoming more organized I fully agree. From AIR I can say its worthy to put some effort in this. There are wonderfully softwares in incoming. If a pkg is poorly presented / build (uploaded) getting disregarded by TUs, we may miss having one more good package.
To make an incoming dir for each TU will not be proper when most contributors don't know the TUs..... don't know whom to upload... in long run all packages will go to contrasutra
The matter is for TUs to know what to test among the 300 incoming. I suggest to use AIR... its a register (not a downloader) to know what are available and TUs can check what's new --> select type of pkg: "Incoming", select "View new pkgs", and press "Submit Search". The description, category and type of application will help TUs to choose instead of going through the incoming-ftp or "New package" forum.
http://bliss-solutions.org/archlinux/incoming/
PS.
Daily I am running one script checking what are new and recognizes the changes between AIR data and ftp repos (incoming and TUR).

I take it you like the AIR?

Personally, I think a database is overkill. Since each TU would only manage his or her folder, it would be very easy to sort through things. Plus, there would be naming requirements (and things would be auto deleted if they didn't conform). Such as "program.PKGBUILD-1.0" or something. A bunch of text files in a folder are very easy to deal with, no need for a db.

Also, the people don't really have to "know" the TUs, they just see what each TU specializes in (or is good with) and how busy they are. If they see a TU has almost no packages, they'd upload to them. If that TU definitely can't test the package for some reason, he would post on the TUR-Mailing List and see if another TU can. If no TU can test the package (very unlikely), you would email the package contributor.

Honestly, this is simpler than it sounds.

rasat · 2004-04-24 07:15:34

contrasutra wrote:

I take it you like the AIR?
Personally, I think a database is overkill. Since each TU would only manage his or her folder, it would be very easy to sort through things.

Yes, I like the AIR . Not because "I" but anyone who had had made it, because of the usefulness.... to know what packages are available. Same as the official package site does.

Why should incoming packages be treated differently than official? Is it not packages / softwares the most important matter for the users, once Arch base is installed and configured? We can understand the incoming stage but TUR is 99.9% as good as official.

In general, I have said earlier about Linux, in Internet there are very little info about softwares compared to configure and hardware matters. I have seen too many users installing Linux and later asking what can it do when not knowing what programs are available. In this regard Windows does better informing more about softwares than configure and hardware issues. No doubt it makes money out of it but thats's not the point.

When the incoming system was improved, TUR introduced, I felt Arch will make a difference as a distro... dealing both with hardware/config and package matter to everyone's satisfaction. But there is a yes and no. Yes in the sense new packages are created. And no in the sense many users don't know what are uploaded in incoming except a post posted in "New Package" forum (some of the TUs don't even inform). When moved to stagging a list is published in the weekly news letter (names only). At final stage, some of the packages are moved in official with a small note in home page (a name link only).

In my opinion, many packages get missed / unnoticed by the users (majority) with the current incoming system when they don't know what's going on. FreeBSD has an option for users to receive an email of list of new packages, upgrades and info. There is the AL's home page package list to search. But currently if we want to find a package e.g in System category, we have to go through 150 packages. With a search keyword, how may users know what to type.... is it "archiving", "archiver" or "backup" for Archiving applications?

To conclude. Here TUs want the incoming to be organized, so why only for the TUs? Why not also include for the users benefit as well?... organize / re-think about the whole package system?

Also, the people don't really have to "know" the TUs, they just see what each TU specializes in (or is good with) and how busy they are.

How will a user / contributor know what a TU is specialized? .... by reading the name of the packages alone??? Example, package names "cjc" and "aldo". What type of application are they? The point is, we should not assume about the users but provide them a means of knowing.

tpowa · 2004-04-24 11:50:56

i think too the database is a good thing because the packages won't appear if you search them on the main homepage

there should be a link on the main page to the database

the main page should be more clear for a new user
it's quite difficult to find the tur repos and for what they are made
something like "more packages are available in tur's" would be enough

i also agree only to post the PKGBUILDS so bandwith can be saved
is it possible to add automatically a new thread to new packages if a new entry is done so everybody can post comments to the package, one of the main problems as tur i can only speak for me but feedback should be better "a works for me" would be enough for example good feedback is also important
another point there is a tur mailinglist that's okay but i would suggest to add a tur forum where the tur packages can be discussed or announced

i don't think it's a good idea to let the users upload to specific tur's.
for example everybody uses different software, i can build them but i don't know how the programs should work etc. (if i would build a new compiler thing or so i can't test that stuff because i don't know much about these programs) then you have to say tur x builds gnome apps tur y kde apps and so on.

linfocito · 2004-04-24 21:23:18

1. I don't like having to download big packages to test them. I can't use the package anyway (dangerous and I have to test rebuilding), so it's wasted time/bandwidth. I think we should convert the system to a PKGBUILD-only system. People can tar up the directory that has the PKGBUILD, .install, patches, etc.

If the other Contrasutra´s suggestions need more discussion, the first one should (IMHO) be implemented as soon as possible.
I see no point in having the pkg.tar.gz files, once INCOMING is about proposing packages to be on extra someday, and that can´t be acomplished without thoroughly testing (both by TUR and end-users), necessarily including build tests (in as many machines as possible).

mcubednyc · 2004-04-24 21:51:21

linfocito wrote:

If the other Contrasutra´s suggestions

There are two Contrasutras??

linfocito · 2004-04-24 22:23:18

Apologizes for my confusing English (I speak Portuguese in my day-by-day) :oops: (Praises for the international fame of Archlinux)

I mean: "The Contrasutra´s other suggestions". (but maybe two contrasutras could be a good idea )

contrasutra · 2004-04-25 04:02:57

OK, I don't really feel there's a big need for the AIR, but most importantly, I don't want to add ANOTHER thing TUs have to do.

Right now, we've got to do all this to get a package in the repos:

Download files from INCOMING
test, fix, rebuild
add/commit the PKGBUILDs in the trunk/ svn repo
add/commit the PKGBUILDs in the tags/ svn repo
use sfs to add the pkg.tar.gz
use ssh to rebuild the pkg database

That's six steps, using three different systems (svn, sfs, and ssh) when it used to be three. I can handle this, but honestly, I don't really want to add any more steps to this.

So if you want to maintain the DB yourself (or with help from other volunteers), that's fine, but I really don't feel like cataloging the pkgs too.

How will a user know what a TU specializes in? There'll be a little "profile" for the TU either on the TUR website or in their INCOMING folder explaining what they're best/most experienced at testing.

rasat · 2004-04-25 05:00:00

contrasutra wrote:

I don't want to add ANOTHER thing TUs have to do.

I got your point. For your information I have taken this in account, to not add any additional work. AIR has one script doing everything automatically, except for the first entry which is done by the contributor. The script automatically modifies the database when TUs have moved packages from incoming to TURs, when moving pkgs between the TURs, and when moved to official.

AIR reduces some of the work of TUs:
1. Instead of downloading files from incoming, PKGBUILDs can be copied/pasted from "New package" forum by checking for new packages in AIR.

2. If we want, we can ask the contributors to copy the PKGBUILD in AIR and with the help of a script upload it to TUR, or wherever they want.

This reduces two steps out of six and there is also no additional work for the contributors except they have to add the description, category and type of application..... this anyhow someone has to add when a package goes to official.

There'll be a little "profile" for the TU either on the TUR web site or in their INCOMING folder explaining what they're best/most experienced at testing.

The script can also be made to read the profile and send the PKGBUILDs to assigned repos. We are here discussing about organizing the incoming, I am suggesting we automate it has as far as possible?

tpowa · 2004-04-25 06:56:38

contrasutra wrote:

Right now, we've got to do all this to get a package in the repos:
Download files from INCOMING
test, fix, rebuild
add/commit the PKGBUILDs in the trunk/ svn repo
add/commit the PKGBUILDs in the tags/ svn repo
use sfs to add the pkg.tar.gz
use ssh to rebuild the pkg database

sfs is not really needed i use ssh for uploading the package too

the database itself is not too much work
mostly it's just cut and paste from the PKGBUILD

why not using the database for storing PKGBUILDS or perhaps linked to the tur PKGBUILDS?

the entries could be automated because it's just simple cut and paste

automatic notifications about new packages is also a great idea

i3839 · 2004-04-25 12:07:37

Speaking for myself, if I must choose between the following two:

1) After making a new package, find from somewhere a TU willing to add my package to the TU's TUR, then wait till that package becomes official.

2) After making a package, just post it on the forum so that I can get quick feedback, and add it to some database so people can find it. Then wait for an uncertain (long?) time till it's official, but in the meantime I have full control over my package.

Option 1 looks like dropping stuff in a black hole to me, option 2 looks a bit more cheerfully. With 1 you put all the load on the TU's, and add an extra communication line, because the TU will get feedback which will or will not be passed to the package maker. With 2 the package maker stays responsible for the package much longer, and also does all the work. What TU's normally would do, initial testing of the package, can be done by the community instead of a few randomly selected people. 2 is only better than 1 if the package maker knows how to make proper packages, but I think that people make packages for applications they use and know, while for a TU it's just another application.

The trusted user idea is fine and well, but either it's for personal repositories, or it's the gathering place for all unofficial packages made by selected trusted users, and then there should be just one global TUR instead of one for each TU. The question is by who the trusted users are trusted. If it's by the Arch developers, they can as well be just package maintainers. If it's by other users, then those users can choose for themselves who they trust (and consequently everyone should be able to become a TU).

I think two stages is enough. In the first stage it's made sure that the package works, this after feedback from users trying it out. In the second stage the pgkbuild and the package are examinated by an official package maintainer, and if it's safe it becomes an official package.

Only criteria for a package to become official should be that it's safe (no malicious or illegal packages), that it works and that there is a maintainer for it. If the new official package comes into the unstable, testing or extra repository doesn't matter, that's part of managing the official packages, not how new packages become official.

FoPref · 2004-04-26 09:35:37

I don't know how far AIR can go, but I would like to suggest just something like a PackageTracker.

Like a BugTracker, everybody submits what he is pleased to. Informations like Categorization, Severity and so on can be added by TUs. So if there's a TU kind enough to take a package under his hood, it just gets assigned to him...
Packages also can be refused and so on like a Bug that's closed. TUs don't have to email through half the world but can easily comment on the trackers interface as the package submitter also is able to to communicate..

Most important thing on it for me as not-TU (hope to be one in future is that I get feedback. Instead of my package lying around on an anonymous ftp site, thats frustrating. with such an interface, I can easily check if there's some progress OR the package is rejected.
On the other side, sorting packages out and adding information about them is an easy and not so much time consuming job. After this is done, it gets more attracting to any TU to build this or that package he is perhaps self-interested in and so on...

cu
Ford Prefect

Mork II · 2004-04-26 15:31:55

Ok, I don't build that many packages (thanks to all your hard work!) so feel free to ignore my ponderings...

The point of incoming, staging and testing is to:
1. Make it possible for users to contribute packages to Arch.
2. Make sure packages are well tested and reasonably stable before beeing included in i]base[/i] or extra.

I think this process could be automated far more than has been suggested so far. At the very least it needs to be more streamlined. I will try to outline a solution in three steps:

Step 1 - Streamlining
New packages should pass two respitories before beeing sent to either base, extra or unstable.

Incoming
PKGBUILD-files of new packages should be sent here, by both users and developers. Users wishing to get a package faster into to the respitories can test building these and submit early bug reports. When a PKGBUILD has been tested by a few different people without failing (see step 2) a binary package can be created (by a developer) and submitted to testing.

Testing
Binary packages are submitted here by developers (not users, no one in there right mind will whish to test binaries from unknown people). When these packages have been tested by a few people (see step 2) they can be moved to either base, extra or unstable .

You will noice that I do not mention Trusted Users in this scheme at all. My suggestion is that the Trusted Users simply are made developers with responsiblity for package submitting system. Senior developers should still be the ones to decide where a package should go after testing.

Step 2 - Testkpkg
Sending in packages by ftp and annoncing them in the forum (also getting bugreports in the forum) seems a bit clumsy to me. I suggest building an application like makepkg, but for submitting and testing packages. Lets call it testpkg.

Testpkg should be able to perform the following operations:

testpkg -submit foo.PKGBUILD
Sends a PKGBUILD to incoming.

testpkg -incoming
Displays a list of PKGBUILDs in incoming, how many times they have been tested and the number of bugs filed against them.

testpkg -testing
Displays a list of packages in testing, how many times they have been tested and the number of bugs filed against them.

testpkg -build foo
Downloads PKGBUILD for foo, creates folder ~/pkg/foo and runs makepkg. It then ask the user a few questions:
- Does the package get built?
- Does it install?
- Does the application work?
When this is done testpkg sends a report back to the Arch server (a database that can store this data would be optimal, otherwise sending it to a mailinglist would suffice), perhaps including user data (see step 3).

testpkg -test foo
Downloads foo from testing and asks the user if:
- It installs?
- Runs?
Sends back feedback to Arch server.

Of course testpkg could be made to run as root, then it could use pacman to install packages as well...

Step 3 - Making use of ArchStats
The ArchStats project would be ideal for creating a bit of friendly competition between users. Testpkg could be made to submit info about it's operations to the ArchStats database where data about the number of PKBUILDS a user has submitted, rate of success, and number of PKBUILDS and packages a user has tested could be stored.

Also when sending in responses via testpkg system information from ArchStats could be submitted to make it easier to find conflicts with particular packages etc.

ArchStats can probably be used some way, even if testpkg is deemed too much of a hassle to implement.

Since I can't program (yet) I can't take these ideas much further...

Xentac · 2004-04-27 17:42:14

I like Mork II's ideas very much (except maybe the ArchStats stuff. That part's only mildly helpful and could be implemented later).

I especially like the testpkg script. Does anyone want to throw something together for this? I'm pretty sure all you'd need is bash to implement it. Oh yeah and make sure it's got a sane filenaming scheme

I'd be willing to offer hints and general direction (and a subversion repository, if you want!) to anyone willing to implement testpkg.

Xentac · 2004-04-27 23:49:22

Ok... reading the testpkg description more it's not exactly what I want.

I want something simpler to automate uploading, downloading, listing, and (perhaps) building packages in incoming. That's it.

Xentac · 2004-04-28 01:33:46

Through the magic of technicolour testpkg has been created.

https://xentac.net/svn/arch-tools/testpkg/trunk/testpkg

It should work well enough for just about everything... I had to write the damned thing though, cause no one was willing to step up

Use it all you like, it should work great.

I'll probably put it in a package later on tonight. Then we'll have to redefine the standards around package uploads... and then we get to start deleting packages from incoming with a vengence!

PS: if you have a package in incoming currently, don't re-upload it, I will be able to convert them all when I make time.

Arch Linux

#1 2004-04-22 19:07:22

Suggestions for INCOMING

#2 2004-04-22 19:14:35

Re: Suggestions for INCOMING

#3 2004-04-22 20:20:12

Re: Suggestions for INCOMING

#4 2004-04-22 21:26:54

Re: Suggestions for INCOMING

#5 2004-04-22 23:42:34

Re: Suggestions for INCOMING

#6 2004-04-22 23:42:53

Re: Suggestions for INCOMING

#7 2004-04-23 04:09:33

Re: Suggestions for INCOMING

#8 2004-04-23 14:19:45

Re: Suggestions for INCOMING

#9 2004-04-23 18:39:19

Re: Suggestions for INCOMING

#10 2004-04-23 19:02:38

Re: Suggestions for INCOMING

#11 2004-04-23 19:34:24

Re: Suggestions for INCOMING

#12 2004-04-24 07:15:34

Re: Suggestions for INCOMING

#13 2004-04-24 11:50:56

Re: Suggestions for INCOMING

#14 2004-04-24 21:23:18

Re: Suggestions for INCOMING

#15 2004-04-24 21:51:21

Re: Suggestions for INCOMING

#16 2004-04-24 22:23:18

Re: Suggestions for INCOMING

#17 2004-04-25 04:02:57

Re: Suggestions for INCOMING

#18 2004-04-25 05:00:00

Re: Suggestions for INCOMING

#19 2004-04-25 06:56:38

Re: Suggestions for INCOMING

#20 2004-04-25 12:07:37

Re: Suggestions for INCOMING

#21 2004-04-26 09:35:37

Re: Suggestions for INCOMING

#22 2004-04-26 15:31:55

Re: Suggestions for INCOMING

#23 2004-04-27 17:42:14

Re: Suggestions for INCOMING

#24 2004-04-27 23:49:22

Re: Suggestions for INCOMING

#25 2004-04-28 01:33:46

Re: Suggestions for INCOMING

Board footer