Arch Linux

ataraxia

Member

From: Pittsburgh

Registered: 2007-05-06

Posts: 1,553

The AUR webapp needs our help!

Last night I was looking at AUR's flyspray, and I saw two things that concern me:
- The intro paragraphs for this project say that it's in "maintenance mode", and "no new features are planned".
- There are several very KISS, very popular, and very good, ideas posted in the feature requests there, which have been stalled for some time.

Looking at the code itself, I am worried about these issues:
- The READMEs and other docs in the tree are so old and crusty, that I can't make much of them.
- The code itself is a bit scary in places, most notably the cryptic variables in pkgfuncs.inc.
- There's a lot of community-specific stuff in there that impedes easy development now that's it's dead code. There's even still older stuff in there that appears to date from the TUR days.

The AUR is an important part of the overall Arch experience. It needs attention from us. Just consider how much nicer it could be if:
- You could get an email notification when a package is updated. (FS 7442) Who needs all those "AUR update checker" utilities when AUR can do that itself?
- You could search and/or sort to find packages you voted on. (FS 5401 and 13643) Votes would be worth a lot more if we could actually see if they were right. Also, this would be a very cheap way to implement a "dashboard" showing all the packages you care about.
- AUR could handle "clever" bash-constructions in the PKGBUILDs? (FS 15043)
- There were some way to keep users from uploading/posting junk? (FS 12902 and 15261, and the thread going on now in aur-general about the Chromium package mess)

It looks to me, based on the low level of traffic on the aur-dev mailing list, that not very many people actually hack on AUR, and that there is far too much work for them to get through it all. This probably leads to the AUR-devs feeling overwhelmed and as if they're doing a thankless job - after all, it works that way for me on my day-job, and that's in spite of actually getting paid for that! I can see several reasons why manpower might be low:
- Community members don't know the AUR codebase is easily available to them (http://projects.archlinux.org/?p=aur.git)
- Community members don't know how to actually test any code changes they might make. Many prospective AUR-hackers surely don't have a LAMP stack available to install AUR on, and we certainly don't have any data to populate our "fake AUR" databases. The stale-doc problem also contributes to this situation.

What's the point of this post? I think there are things the community can do to help the AUR, and there are also things the AUR-devs can do to help the community become more helpful:
- We need to know whether to fix the AUR, or rewrite it. The AUR devs should clarify what it means to the rest of the community for it to be "in maintenance mode". Do you not actually want our patches? Do you wish someone would write a new app on their own and take it off your hands?
- Users of the AUR should give some thought to what they would like out of it. Just because it's "always been that way" doesn't mean it can't change now.
- Someone, either devs or community members, should set up a semi-public "fake AUR" where development can occur. If I had access to such a thing, even I would be working on several patches, and I don't even have a background in PHP!

Consider this incredibly-dash-riddled post a call to arms for the future of the AUR. The AUR is such a great idea in principle, so why can't it be as great in practice as it is in our heads? Look at all the fine work that's gone into AUR-helper-apps. Imagine what things could be like if all that talent and enthusiasm could be channeled directly into AUR itself.

rcoyner

Member

From: Washington D.C.

Registered: 2008-05-16

Posts: 30

Website

Re: The AUR webapp needs our help!

It looks to me, based on the low level of traffic on the aur-dev mailing list, that not very many people actually hack on AUR, and that there is far too much work for them to get through it all. This probably leads to the AUR-devs feeling overwhelmed and as if they're doing a thankless job - after all, it works that way for me on my day-job, and that's in spite of actually getting paid for that! I can see several reasons why manpower might be low:
- Community members don't know the AUR codebase is easily available to them (http://projects.archlinux.org/?p=aur.git)
- Community members don't know how to actually test any code changes they might make. Many prospective AUR-hackers surely don't have a LAMP stack available to install AUR on, and we certainly don't have any data to populate our "fake AUR" databases. The stale-doc problem also contributes to this situation.

I would actually say that a big part of the reason why there hasn't been a lot of development effort into the current AUR is because there have been attempts at a re-implementation of the current system. Nobody wants to contribute to a codeset that is going to be re-written. And re-implementations have been attempted because the current design of the system is... not flawed, but would be greatly beneficial if started from the ground up. Although it is possible to implement neat features like an API, VCS integration and a proper PKGBUILD parser into the current system, it would be through a lot of band-aid patches. I'm actually in the process of writing a PKGBUILD parser in Lex/Yacc so I will hopefully be releasing that soon.

Anyway, most of these re-implementation efforts have failed. So until somebody consistently works on a re-implementation and gets it right, the AUR is probably going to look pretty stagnant. That's my 2c at least. I'm curious as to what Loui has to say since he's our resident AUR expert.

Peasantoid

Member

Registered: 2009-04-26

Posts: 928

Website

Re: The AUR webapp needs our help!

Probably not going to be bothered with, but worth a try...

I took a brief glance at the codebase and while it's pretty good for a PHP application, I'm not sure if that's the best language for such a thing. You could do a lot worse (.NET, for one), but you could also do a lot better. At this point I'll say Python, just because I've managed to do some truly cool things with it. You also don't need to dick around with php.ini, which is always a plus.

Of course, this is only relevant if the application actually is going to be rewritten.

// Wow, suppressing my inner PHP-hater took a lot of work. Here's what I actually think:
// AAAAAAARRRRRRRRRRRGGGHHHHHHH NOOOOOOOOOO *sob*

Last edited by Peasantoid (2009-09-27 09:55:40)

Stefan Husmann

Member

From: Germany

Registered: 2007-08-07

Posts: 1,391

Re: The AUR webapp needs our help!

The current parts of AUR 2 are written in Python/Django. Search for AUR2 in the wiki for more.

louipc

Member

Registered: 2006-10-09

Posts: 85

Re: The AUR webapp needs our help!

Maintenance mode means that I will review and accept patches.
I have no plans to develop new features in the current implementation of the AUR.

If you do want to contribute, a lot of the setup/data issues you mention have been
considered and there's decent documentation and scripts to help you.
It's certainly better than when I started.

Why is the AUR in maintenance mode?
I don't believe the AUR should be primarily a web app.
The problem I see with previous attempts to rewrite the AUR is that they were the
same old AUR, just in a different programming language.
It's like if someone were to rewrite CVS in lisp. There's really no point. It's still dumb old CVS.

So most of my energy (when I find the inspiration) is used in experimenting and thinking
up ways to change the system.
Be aware that I'm no programming guru that can write an amazing app over a week.
I started hacking on the AUR because there were a few nagging bugs.
My patching went overboard and now I'm a maintainer.

I recommend that you subscribe to the aur-dev mailing list if you're interested in getting
involved, since development is mostly discussed there.

Thanks for your interest.
Cheers.

ataraxia

Member

From: Pittsburgh

Registered: 2007-05-06

Posts: 1,553

Re: The AUR webapp needs our help!

Why is the AUR in maintenance mode?
I don't believe the AUR should be primarily a web app.
The problem I see with previous attempts to rewrite the AUR is that they were the
same old AUR, just in a different programming language.
It's like if someone were to rewrite CVS in lisp. There's really no point. It's still dumb old CVS.

So what do you think it really should be? A "repo" accessed by a sort of super-yaourt? Something more like the ABS tree, or aur-sync?

Taking it away from the web-app focus seems fine in principle, but in practice will probably be even harder to please the majority of the users with than the current model.

I will sub to aur-dev, though I probably won't say much.

Peasantoid

Member

Registered: 2009-04-26

Posts: 928

Website

Re: The AUR webapp needs our help!

The current parts of AUR 2 are written in Python/Django. Search for AUR2 in the wiki for more.

Hell, I might get involved with that. (Useful for teaching myself the ropes...)

Xilon

Member

Registered: 2007-01-01

Posts: 243

Re: The AUR webapp needs our help!

Why is the AUR in maintenance mode?
I don't believe the AUR should be primarily a web app.
The problem I see with previous attempts to rewrite the AUR is that they were the
same old AUR, just in a different programming language.
It's like if someone were to rewrite CVS in lisp. There's really no point. It's still dumb old CVS.

Indeed, it probably shouldn't be primarily a web app, however there probably should be a central database (for Archlinux purposes - it should be possible to set up your own) and a web app is perfect for that. The data store and web app do not have to be tightly coupled. The web app could simply be just another client communicating via an API, but I think a web app is still a good idea. The main site has a web front-end for the repositories as well.

My efforts to write AUR2 (wiki, code) have lately been focused towards making a generic package catalogue web app, much like the main site's, but initially working with PKGBUILDs. AUR2 has come a long way though, and little needs to be worked on, in order to get it up to scratch, relatively speaking.

I'm actually in the process of writing a PKGBUILD parser in Lex/Yacc so I will hopefully be releasing that soon.

That's great to hear. I was planning on making a proper PKGBUILD parser as well, but ended up making a primitive version in Python . Post the code up as soon as you can, I'm very interested!

Last edited by Xilon (2009-10-19 11:54:07)

andrewjames

Member

From: Canada, Ontario, Toronto

Registered: 2009-06-09

Posts: 19

Website

Re: The AUR webapp needs our help!

I am a relatively new user of arch linux.  I have seen the repository AUR (hypertext), it is some confusion, disorderly, unattractive.

I anticipate a new program for the community repository (AUR).  Perhpase there is a line for someone to clarify the uses of AUR compared to pacman 'community'.  I think AUR is undeniably a community, of users.

I have subscribed to some mail lists development to collabourate as I can.

Thank you, ataraxia, to coordinator.

---

andrew james

legolas558

Member

Registered: 2009-09-08

Posts: 97

Re: The AUR webapp needs our help!

looks really interesting!

---

.-.   ,---,---.--.
| |__  \ \ \ \`//.
`----'`--'`--'`--'

djszapi

Member

From: Cambridge, United Kingdom

Registered: 2009-06-14

Posts: 1,439

Website

Re: The AUR webapp needs our help!

Can we know aomething about YACC/LEX parser for PKGBUILDs ? Can we help in it for you ?
I agree with you, guys AUR is important part of Archlinux. This is the first interface for the most of newbies to contribute e.g.
I agree with louipc in that no need to be a web application. All the other core projects on arch are console based.
Some example: pacman, abs, dbscripts, devtool, pkgtools, aif(Archlinux installer project), namcap, netcfg, mkinitcpio, archiso, archboot, etc..etc.. It's no need really to be a web application.

Btw, I started to implement a new AUR Manager, you can see here the project
http://developer.berlios.de/projects/aurman/
mailing list:
https://lists.berlios.de/mailman/listinfo/aurman-dev
the code base:
http://git.berlios.de/cgi-bin/gitweb.cg … ;a=summary
Any help would be nice for me

Some discussion is happening on aur-dev@archlinux.org and #archlinux-aur on freenode, we will see what will happen, but I try to establish that api, backend that would be good for all frontend.

The most difficult task of the whole thins is the PKGBUILD parsing.

Last edited by djszapi (2009-10-24 09:44:39)

---

http://lpapp.blogspot.com

extofme

Member

From: here + now

Registered: 2009-10-10

Posts: 174

Website

Re: The AUR webapp needs our help!

i agree that the AUR, at least IMO, is one of the most attractive things about arch linux.  i have bounced around between many distros over many years, and have settled with arch for a variety of reasons; the AUR is near the top.  coming from the persepective of a new user to arch but an experienced developer and linux enthusiast, i was able to contribute a package [zarafa-server] within a week of starting with arch, and i thouroughly support and encourage its expansion and development.  with that thought, i do not think its a good idea to restrict the AUR too much, as in my case i probably never would have contributed at all if i had needed to wait for some kind of knightly dubbing by the royalty of arch forums.

take the chromium builds for example.  packages could be better categorized to check for duplicates by diffing the sources array in the PKGBUILD, scanning for duplicate sourcing URLs/fragments amongst packages.  in this new system, as another interested AUR/chromium user, i could submit patches against the current PKGBUILD incarnation and wait for the responses from the current maintainer.  meanwhile, i add myself to a list of interested maintainers for the chromium package.  should the package cross a "stagnation threshold", i could potentially be selected as the new maintainer.  should the package be flagged as "uncooperative" or "rouge", the package's users could elect a new maintianer from the list of interested parties.  all of this merging/patching could easily be handled by git in the backend.  something of this design would allow new users the freedom to contribute without blockades, and allow those truly devoted to quality and responsiveness to be recognized and gain the merits of an unofficial TU of sorts.

as for parsing PKGBUILD, i honestly see no reason for that at all.  i have been using virtualization and containment techniques for many years, and kernel namespace solutions like LxC are extremely robust and secure.  create a cheap "analysis" container, source the PKGBUILD, and get the variables you need; parsing will never be 100% and adds unecessary complexity.

an AUR daemon is what is needed;  daemon can respond to a API calls from a web-based frontend, and also new tools built for the console.

on a side note, i would love to see unified auth; having separate accounts (even though mine are all the same) for AUR/BBS/flyspray is rather annoying.  we need single sign-on here.

+1 for all of it in Python

---

what am i but an extension of you?

rcoyner

Member

From: Washington D.C.

Registered: 2008-05-16

Posts: 30

Website

Re: The AUR webapp needs our help!

as for parsing PKGBUILD, i honestly see no reason for that at all.  i have been using virtualization and containment techniques for many years, and kernel namespace solutions like LxC are extremely robust and secure.  create a cheap "analysis" container, source the PKGBUILD, and get the variables you need; parsing will never be 100% and adds unecessary complexity.

I feel like this is a complicated solution to a much simpler problem, as well as it being the wrong tool for our problem. Admittedly I don't know a whole lot about LxC but it sounds to me like it's more of a general purpose management solution for application workload isolation and virtual private servers... Can you elaborate as to how this would help us?

an AUR daemon is what is needed;  daemon can respond to a API calls from a web-based frontend, and also new tools built for the console.

+1

on a side note, i would love to see unified auth; having separate accounts (even though mine are all the same) for AUR/BBS/flyspray is rather annoying.  we need single sign-on here.

+1 for all of it in Python

Unified authorization is a little tricky to achieve with the current range of products we use. If we switch over to Django apps, it would be a lot easier...

http://djangobb.org/ is a potential alternative to the current BBS. Not sure how to replace Flyspray.

Xilon

Member

Registered: 2007-01-01

Posts: 243

Re: The AUR webapp needs our help!

i agree that the AUR, at least IMO, is one of the most attractive things about arch linux.  i have bounced around between many distros over many years, and have settled with arch for a variety of reasons; the AUR is near the top.  coming from the persepective of a new user to arch but an experienced developer and linux enthusiast, i was able to contribute a package [zarafa-server] within a week of starting with arch, and i thouroughly support and encourage its expansion and development.  with that thought, i do not think its a good idea to restrict the AUR too much, as in my case i probably never would have contributed at all if i had needed to wait for some kind of knightly dubbing by the royalty of arch forums.

Indeed, it's a great system. Contributing to Debian for instance is a nightmare. I think I spent like 3 days reading the documentation for packaging and still wasn't able to package anything. Luckily there was checkinstall .

take the chromium builds for example.  packages could be better categorized to check for duplicates by diffing the sources array in the PKGBUILD, scanning for duplicate sourcing URLs/fragments amongst packages.

I don't think this is a good idea. Contributers should spend the extra 5 minutes to search for the package before even starting packaging themselves.

in this new system, as another interested AUR/chromium user, i could submit patches against the current PKGBUILD incarnation and wait for the responses from the current maintainer.  meanwhile, i add myself to a list of interested maintainers for the chromium package.  should the package cross a "stagnation threshold", i could potentially be selected as the new maintainer.  should the package be flagged as "uncooperative" or "rouge", the package's users could elect a new maintianer from the list of interested parties.  all of this merging/patching could easily be handled by git in the backend.  something of this design would allow new users the freedom to contribute without blockades, and allow those truly devoted to quality and responsiveness to be recognized and gain the merits of an unofficial TU of sorts.

Indeed. I wanted a git backend as well. It should be easy enough to run a db-updating script on push. The added benefit is that maintainers can just have a repo locally and `git push`, rather than having to go to the web interface and uploading manually, or using a standalone client. AUR2's database schema currently supports multiple maintainers, but I'm not sure if it's a good idea to actually have that feature.

as for parsing PKGBUILD, i honestly see no reason for that at all.  i have been using virtualization and containment techniques for many years, and kernel namespace solutions like LxC are extremely robust and secure.  create a cheap "analysis" container, source the PKGBUILD, and get the variables you need; parsing will never be 100% and adds unecessary complexity.

Would this be efficient? Sounds a lot like chrooting, etc. An AUR could potentially handle many upload requests, and would have to parse PKGBUILDs quickly.

an AUR daemon is what is needed;  daemon can respond to a API calls from a web-based frontend, and also new tools built for the console.

You may not realise it but this is the current case (well, not exactly). An web server is a daemon, and the web frontend responds to API calls. Should daemon be separated from the frontend? I'm not sure, but I'm leaning towards "no".

What would the daemon do? It seems to be that it would simply replicate the functionality of a web server and add a few features of AUR.

on a side note, i would love to see unified auth; having separate accounts (even though mine are all the same) for AUR/BBS/flyspray is rather annoying.  we need single sign-on here.

While it would be tricky to unify the system, it should be possible to allow for OpenID logins everywhere, thus having a single login (you'd still have to login multiple times though).

extofme

Member

From: here + now

Registered: 2009-10-10

Posts: 174

Website

Re: The AUR webapp needs our help!

I don't think this is a good idea. Contributers should spend the extra 5 minutes to search for the package before even starting packaging themselves.

i do agree that people should look first... but it cant be depended on.  if we could at least identify what people were trying to accomplish with their PKGBUILD, we could put all chromium builds for example it its own directory/whatever, and when someone tried to upload, maybe we'd have a "did you mean..." type of screen.  even with git console tools, we can have backend scripts that perform this check for git/console uploads

Indeed. I wanted a git backend as well. It should be easy enough to run a db-updating script on push. The added benefit is that maintainers can just have a repo locally and `git push`, rather than having to go to the web interface and uploading manually, or using a standalone client. AUR2's database schema currently supports multiple maintainers, but I'm not sure if it's a good idea to actually have that feature.

i think the multiple maintainer idea is more to support the above.  one person uploads an SCM version, another uses stable version X, etc.  these people should be forcefully grouped together so that others can see what is available and what the differences are in the actual PKGBUILD.  maybe would promote sharing between build for robustness/completeness and maybe promote unification also?  im not sure how this would look exactly but it might be worth some further thought

as for parsing PKGBUILD, i honestly see no reason for that at all.  i have been using virtualization and containment techniques for many years, and kernel namespace solutions like LxC are extremely robust and secure.  create a cheap "analysis" container, source the PKGBUILD, and get the variables you need; parsing will never be 100% and adds unecessary complexity.

Would this be efficient? Sounds a lot like chrooting, etc. An AUR could potentially handle many upload requests, and would have to parse PKGBUILDs quickly.

I feel like this is a complicated solution to a much simpler problem, as well as it being the wrong tool for our problem. Admittedly I don't know a whole lot about LxC but it sounds to me like it's more of a general purpose management solution for application workload isolation and virtual private servers... Can you elaborate as to how this would help us?

these containers are actually really lightweight.  all they really are is a private PID/User/Network and various other namespaces.  see here where we are working to get the necessary options enabled by default in arch kernel:

http://bugs.archlinux.org/task/16715

specifically in that bug is a link about performance:

https://lists.linux-foundation.org/pipe … 21403.html

performance hits are considered negligeble or nonexistent.  the libvirt project has native bindings in python to control LXC, although their are native LXC tools as well, but im unsure of their bindings...  i am using the libvirt library/python to control containers and create them on the fly; its very fast and easy.  you have the option to chroot (actually pivot_root) but it isnt required.

we could use these lightweight containers to source the PKGBUILD.  we wouldnt even need to create/destroy them, we'd have a dedicated container doing one after the other; we wouldnt need to rebuild it for each PKGBUILD.  if we can find a way to just source the PKGBUILD (with some sane limits of course... a PKGBUILD should only take .0000001ish/fast seconds to source) it will reduce complexity in the backend and increase correctness in the frontend.

LXC may not be the way in the end, but somehow enabling simply sourcing the PKGBUILD is i think a better solution to parsing.

You may not realise it but this is the current case (well, not exactly). An web server is a daemon, and the web frontend responds to API calls. Should daemon be separated from the frontend? I'm not sure, but I'm leaning towards "no".

What would the daemon do? It seems to be that it would simply replicate the functionality of a web server and add a few features of AUR.

i guess i would just expect it to be completely separate from the webapp.  i have written PHP/etc webservices, maybe something more like that; where the website is standalone, and makes authenticated requests to the AUR webservice.  console tools could do the same, and they would recieve a standard output like XML or something (i kindof loathe XML.. maybe a pickle/protobuffer or something, would have to be friendly and maybe not language specific).  this would not be daemonized though.  i like the daemon becaseu the AUR machine could just /etc/rc.d/aur start (the AUR is Arch right :-) ), and the daemon would be responsible for setting up a sourcing container and connecting to port X, and standing by for client requests.

in the future, if binary builds of really popular PKGBUILDs were enabled, daemon could create a build container and create binary packs, further closing the gap and accelerating their inclusion into [community]

be tricky to unify the system, it should be possible to allow for OpenID logins everywhere, thus having a single login (you'd still have to login multiple times though).

i like the openid idea, there might be some code out there to store an encrypted cookie or something for persistent login between domains (forum/bugs/aur), but i think just going to unified login would be a good step.

---

what am i but an extension of you?

fogobogo

Member

Registered: 2008-08-24

Posts: 83

Re: The AUR webapp needs our help!

yeah. that AUR is getting old and ugly. Probably everyone has been bothered by one quirk or an other. Then again... it seems to work "just good enough". or bad enough. whatever one thinks.
Since all past attempts to get AUR2 done failed so far, the modus operandi might need a change.
The best I could come up with is kind of a Google Summer of Code thing. a few well-seasoned webdevs sketch a design and play the mentor role whilst some "students" execute it. The reward would be the learning experience and well as a nice entry to the references list. no idea that will ever work or is even a mildly good idea. just my 2c

Xilon

Member

Registered: 2007-01-01

Posts: 243

Re: The AUR webapp needs our help!

I don't think this is a good idea. Contributers should spend the extra 5 minutes to search for the package before even starting packaging themselves.

i do agree that people should look first... but it cant be depended on.  if we could at least identify what people were trying to accomplish with their PKGBUILD, we could put all chromium builds for example it its own directory/whatever, and when someone tried to upload, maybe we'd have a "did you mean..." type of screen.  even with git console tools, we can have backend scripts that perform this check for git/console uploads

This could be handled by simply fuzzy-matching the pkgnames (and tags?), and showing the list at the upload page. How would this be handled in standalone clients though? It's an extra, annoying, step for the contributer.

You may not realise it but this is the current case (well, not exactly). An web server is a daemon, and the web frontend responds to API calls. Should daemon be separated from the frontend? I'm not sure, but I'm leaning towards "no".

What would the daemon do? It seems to be that it would simply replicate the functionality of a web server and add a few features of AUR.

i guess i would just expect it to be completely separate from the webapp.

The web app could be separate, but really, how different is the web app from the server? When implementing the API (commits were lost ) I made it so that based on the extension (.json, .xml, .html), the output was changed. Thus everything was the same, except for the template handling. Whether I (hypothetically) go to http://aur.archlinux.org/packages.html (the .html could be optional and thus default) or http://aur.archlinux.org/packages.json, the same data is presented, just in a different format. Why separate these two when they are logically identical? All that would be achieved is duplication of code, and loss of performance.

Edit: In the case of the Django AUR2, the code base could be separate (two different Django apps), thus allowing for a different installation to be only the "daemon", without the (html) "web frontend". I'm not sure how useful this would be, since the only difference would be the lack of HTML-related resources (templates, CSS, JavaScript, images).

i have written PHP/etc webservices, maybe something more like that; where the website is standalone, and makes authenticated requests to the AUR webservice.  console tools could do the same, and they would recieve a standard output like XML or something (i kindof loathe XML.. maybe a pickle/protobuffer or something, would have to be friendly and maybe not language specific).  this would not be daemonized though.

That is the intended design of the AUR2, the default format for the API would be JSON. Not sure if XML is worth supporting (in Django it's trivial, it's just s/json/xml/).

i like the daemon becaseu the AUR machine could just /etc/rc.d/aur start (the AUR is Arch right :-) ), and the daemon would be responsible for setting up a sourcing container and connecting to port X, and standing by for client requests.

I like the web server because the AUR machine could just /etc/rc.d/httpd start (the AUR is Arch right :-) ), and the web server would be responsible for setting up a sourcing container and connecting to port X, and standing by for client requests.

Again, how is that different from a web server?

LXC may not be the way in the end, but somehow enabling simply sourcing the PKGBUILD is i think a better solution to parsing.

I have started another PKGBUILD parser with Yacc/Lex, like rcoyner's. I made a different one because I thought rcoyner's implementation was too rigid and hardcoded. This parser is a pseudo-bash parser, attempting to comply to IEEE Std 1003.2 (POSIX Shell Standard). It will probably deviate a bit from the standard (especially since it's it seems the standard is sh, not bash) to make it easier and faster to parse. The whole standard will definitely not be supported. Currently only word assignment and (partially) expansion is supported, which covers most PKGBUILDs. Next up is split package support (variables set in functions), and later Pattern Matching Notation  (e.g. ${foo#bar}) and possibly parameter expansion modifiers (e.g. ${foo:-bar}). I will be modifying my python parser (currently equivalent in functionality), to use pkgparse, which will hopefully speed it up .

If this works well, I don't really see a reason for an alternate solution.

in the future, if binary builds of really popular PKGBUILDs were enabled, daemon could create a build container and create binary packs, further closing the gap and accelerating their inclusion into [community]

This would require either multiple machines for each architecture, or chroot environments per architecture. The environment would have to be cleaned up after each build. There are also major security issues, though these might be solved by the container solution you suggested.

Since all past attempts to get AUR2 done failed so far, the modus operandi might need a change.

I wouldn't say all past attempts have failed. My AUR2, while stagnant, is still alive, and once I get some more time, I'll try to get back to it. Everyone is, of course, welcome to contribute . It seems that the "BR" AUR2, however, is dead (used to live at http://aur2.archlinux-br.org/). It was also stagnant for quite some time now (even before my AUR2 was started).

Last edited by Xilon (2009-10-29 10:41:12)

mjunx

Member

From: Chicago

Registered: 2009-11-01

Posts: 17

Website

Re: The AUR webapp needs our help!

Simple daemon idea: just use HTTP as the daemon. One could just use REST and simply use the methods available to HTTP in the first place which tend to work out pretty well in many situations, especially ones where files are simply being sent back and forth.

extofme

Member

From: here + now

Registered: 2009-10-10

Posts: 174

Website

Re: The AUR webapp needs our help!

Simple daemon idea: just use HTTP as the daemon. One could just use REST and simply use the methods available to HTTP in the first place which tend to work out pretty well in many situations, especially ones where files are simply being sent back and forth.

simple web service will work fine in the traditional way the AUR is used, as far as i can tell.  however, it is my opnion that there needs to be a clean separation between the libraries and data structures that implement the AUR entity, and the frontend service that delivers it to clients.

if i want to build an AUR server that is not HTTP based, i should not need to reimplement all the AUR functionality to do so.  i should simply be able to import the AUR library and write tools to work with it.  if the library is too intertwined with the web service, things like this are not possible; we will not be free to experiment with alternative delivery methods and structural hierarchies.

for example, i want to play with the idea of a distributed ABS/AUR, where each package is a branch, and the whole thing can be pulled with a clone.  people can push and pull package variations between each other without ever touching "official" servers.  in this setup there would be nothing but hooks to a [git] repository.  the hooks need to link with a library not a webservice.

i come to this from reading various sources on the unification of linux based packaging systems.  in reality windoze is more "unified" and "distributed", because i can go to any old website, download an exe, at it works pretty much as expected.  however in linux, authors are not able to effectivly publish vanilla packages due to great variation in management systems, and the various custom patches between distros.  i'm digressing, but my point is that there are lots of ways to think about how the AUR can work, and limiting that view to a web server daemon will hinder any experimental ideas.

i do think the web service is a good start, maybe a lightweight CherryPy server.

---

what am i but an extension of you?

Xilon

Member

Registered: 2007-01-01

Posts: 243

Re: The AUR webapp needs our help!

simple web service will work fine in the traditional way the AUR is used, as far as i can tell.  however, it is my opnion that there needs to be a clean separation between the libraries and data structures that implement the AUR entity, and the frontend service that delivers it to clients.

I'm curious how this library would be designed, and what its role would be. AUR is composed of a few elements:

* Data-store (currently a MySQL database)
* PKGBUILD parser
* Glue code, CRUD operations on the database based on PKGBUILD metadata and user interaction
* The view (web frontend, API)

A standalone PKGBUILD parsing library is in the works, as already mentioned. The data-store is obviously separate. The glue code is the heart of AUR. It's what does a majority of the operations, and it's the component that can't really be decoupled any further (Sure, you could abstract the data store and have callback functions or something, but that's overkill).

for example, i want to play with the idea of a distributed ABS/AUR, where each package is a branch, and the whole thing can be pulled with a clone.  people can push and pull package variations between each other without ever touching "official" servers.  in this setup there would be nothing but hooks to a [git] repository.  the hooks need to link with a library not a webservice.

In this example, the data-store is the git repository. The view would be the filesystem itself. The "glue code" would be the git hooks. I don't see a way of re-using any of the AUR glue code for the git repository glue code. They are completely different at a conceptual level (the AUR only allows logged in users to upload packages, only TUs can delete packages, etc. Git repositories don't have these issues).

It seems to me like everyone thinks that AUR2 is a panacea. It's not. It has a specific purpose (to be a central repository of user-contributed PKGBUILDs). If you want to do something else with the concept of AUR (user contributed packages), then by all means do it. The PKGBUILD parser is really the only re-usable thing, and that's being developed as a standalone library. Don't wait around for this magical AUR2 to be finished. It won't help you if you want to make a peer-to-peer repository of PKGBUILDs, or some other, completely different idea.

Note: Yes, I am referring to my implementation of AUR when using the term "AUR2." This is because it's the natural successor of the current AUR (the concept hasn't changed). Your "AUR2" may be completely different, but as I said, the AUR is the AUR.

__void__

Member

Registered: 2009-01-21

Posts: 10

Re: The AUR webapp needs our help!

I'm curious how this library would be designed, and what its role would be. AUR is composed of a few elements:

* Data-store (currently a MySQL database)
* PKGBUILD parser
* Glue code, CRUD operations on the database based on PKGBUILD metadata and user interaction
* The view (web frontend, API)

Looking at Perl's CPAN I've found an interesting approach, CPAN has a different site for the submission and administration of packages. This separation could be done in AUR, i.e: a site specific to search packages and a site for managing the packages. This approach could make the interfaces to the sites richer, as they only have to do "one side" of the AUR workflow.

A standalone PKGBUILD parsing library is in the works, as already mentioned. The data-store is obviously separate. The glue code is the heart of AUR. It's what does a majority of the operations, and it's the component that can't really be decoupled any further (Sure, you could abstract the data store and have callback functions or something, but that's overkill).

I think this is the only thing that can be encapsulated as a library, providing the business logic to interact with the database, i.e, providing functions like searching packages on the storage, CRUD, etc.

The interfaces should use this library instead of mangling the storage data themselves, i.e: the django view for searching packages would call the aur library search function reformat the results and show them in the web.

The same could be done to provide a RESTfull interface (the default service for accessing the AUR).

Maybe this library thing is a bit of overkill, as a web client and a REST api could be done using a django project and django-piston (an app to provide REST interfaces for models). So adding the REST interface to the current AUR2 project is only matter of adding django-piston app and adding some REST views that expose the store models.

The dissadvantage in this approach is that we're tiying the AUR to Django, if AUR provides an abstraction library server and consumer applications could be done in any language that the library provides bindings to.

for example, i want to play with the idea of a distributed ABS/AUR, where each package is a branch, and the whole thing can be pulled with a clone.  people can push and pull package variations between each other without ever touching "official" servers.  in this setup there would be nothing but hooks to a [git] repository.  the hooks need to link with a library not a webservice.

In this example, the data-store is the git repository. The view would be the filesystem itself. The "glue code" would be the git hooks. I don't see a way of re-using any of the AUR glue code for the git repository glue code. They are completely different at a conceptual level (the AUR only allows logged in users to upload packages, only TUs can delete packages, etc. Git repositories don't have these issues).

There the library approach will help, providing high level functions that take the job of accessing the storage. Later the filesystem layer could be switched and the client apps using this library won't have to change a line of code.

It seems to me like everyone thinks that AUR2 is a panacea. It's not. It has a specific purpose (to be a central repository of user-contributed PKGBUILDs). If you want to do something else with the concept of AUR (user contributed packages), then by all means do it. The PKGBUILD parser is really the only re-usable thing, and that's being developed as a standalone library. Don't wait around for this magical AUR2 to be finished. It won't help you if you want to make a peer-to-peer repository of PKGBUILDs, or some other, completely different idea.

I've surfing your AUR2 project, and as I said before the API design that you propose on the AUR2 wiki page can be easily implemented on top of your project by using django-piston (or custom code).

Also it would be good to investigate further interaction with the arch main site (I'm trying to help in the development of that project too). As it would be really nice to have a unique userbase for all the Arch related services (AUR, Wiki, Forum, Bugtracker).

djszapi is trying to rebirth the development of this project, I volunteer to help in the Django (frontend, api) coding. Xilon are you up to continue your work with this project?

__void__

Member

Registered: 2009-01-21

Posts: 10

Re: The AUR webapp needs our help!

yeah. that AUR is getting old and ugly. Probably everyone has been bothered by one quirk or an other. Then again... it seems to work "just good enough". or bad enough. whatever one thinks.
Since all past attempts to get AUR2 done failed so far, the modus operandi might need a change.
The best I could come up with is kind of a Google Summer of Code thing. a few well-seasoned webdevs sketch a design and play the mentor role whilst some "students" execute it. The reward would be the learning experience and well as a nice entry to the references list. no idea that will ever work or is even a mildly good idea. just my 2c

The AUR is one of the best features of ArchLinux. If the main devs promote and backup the idea that the AUR must be reimplemented the Arch community will be more interested on thinking/proposing new AUR implementations, this could be an "Arch sponsored project" with the guiding of some TU, and the community making the proposals and later implementing the selected one.

I think that ArchLinux should try to provide the best tools for the community to contribute to it, and the AUR is the main channel (and differential with other distros) of participation, so the AUR infraestructure should be taken as a really important part of the distro, the contents of the AUR aren't official, but the tools provided should be, and ArchLinux should try to give the best implementation it cans.

What do you think of making a 'google SOC' like project parented by some Archlinux main devs, writing an abstract of the project scope and accepting design submissions by the community. Then the implementation will be done by some community members with the approval of the TUs leading the process.

Xilon

Member

Registered: 2007-01-01

Posts: 243

Re: The AUR webapp needs our help!

Looking at Perl's CPAN I've found an interesting approach, CPAN has a different site for the submission and administration of packages. This separation could be done in AUR, i.e: a site specific to search packages and a site for managing the packages. This approach could make the interfaces to the sites richer, as they only have to do "one side" of the AUR workflow.

As far as Django goes, this separation would be quite simple. The "aur" app would simply have to be split into "aur" and "aur_admin" (or whatever is appropriate), with "aur_admin" using models from the "aur" app. Where does uploading packages fit into this? The admin part? The views are already separate in that case.

I think this is the only thing that can be encapsulated as a library, providing the business logic to interact with the database, i.e, providing functions like searching packages on the storage, CRUD, etc.

The interfaces should use this library instead of mangling the storage data themselves, i.e: the django view for searching packages would call the aur library search function reformat the results and show them in the web.

So what you're proposing is to indeed abstract away the package management logic? So to get a package, instead of doing package = Package.objects.get(name="foo"), you want to do package = db.packages.get(name="foo") (where db is an instance of Aur.Database or something). Where the db.packages.get() function in turn either retrieves information directly from the database, or calls Package.objects.get() anyway (and converts a Django "Package" model into whatever structure the AUR library uses)?

The AUR library would likely have to be in C (easiest to bind into other languages), so to use the Django API you'd have to implement the backend class in the Python (requiring Python bindings):

class DjangoBackend(Aur.DatabaseInterface):
    def get(name):
        return djangoPackageToAurPackage(Package.objects.get(name=name) # except better

This is a huge decrease in efficiency and flexibility (not to mention it would be ugly if implemented in C), just so that two completely different projects can use the same library, which likely won't be that useful anyway. The back-end will still be a large part to implement, and that's left up to the user. Maybe if it's abstracted well, with a flexible design, it wouldn't be so bad. On first glance, it's not a good idea.

The dissadvantage in this approach is that we're tiying the AUR to Django, if AUR provides an abstraction library server and consumer applications could be done in any language that the library provides bindings to.

In theory yes, but providing those bindings won't always be an easy task. There will also be a lot of "translating" between the actual backend, and the AUR library. I'm not sure if it would be work the portability.

Also it would be good to investigate further interaction with the arch main site (I'm trying to help in the development of that project too). As it would be really nice to have a unique userbase for all the Arch related services (AUR, Wiki, Forum, Bugtracker).

A common login would be nice, and being able to get information about dependencies from the main repositories would also be great. However, I doubt this will happen (well, maybe the login will), as the AUR is supposed to be separate. What I was planning to do is make AUR2 flexible enough to also be a frontend for the main repositories, effectively succeeding AUR and the front page catalogue (the "packages" app from archweb).

djszapi is trying to rebirth the development of this project, I volunteer to help in the Django (frontend, api) coding. Xilon are you up to continue your work with this project?

I hope so, I just don't know when I'll have the time to do so. I am making progress on the parser though. Here's some good news:

$ cat PKGBUILD

pkgname=('llvm' 'clang' 'clang-analyzer')
pkgver=2.6
pkgrel=3
arch=('i686' 'x86_64')
makedepends=('gcc-libs' 'python')
source=(http://llvm.org/releases/$pkgver/$pkgname-$pkgver.tar.gz http://llvm.org/releases/$pkgver/clang-$pkgver.tar.gz)
md5sums=('34a11e807add0f4555f691944e1a404a' '09d696bf23bb4a3cf6af3c7341cdd946')

package_llvm() {
  pkgdesc="Low Level Virtual Machine"
  url="http://llvm.org/"
  license=('custom:University of Illinois/NCSA Open Source License')
  depends=('perl')
}

package_clang() {
  pkgdesc="C language family frontend for LLVM"
  url="http://clang.llvm.org/"
  license=('custom:University of Illinois/NCSA Open Source License')
  depends=('gcc-libs')
}

package_clang-analyzer() {
  pkgdesc="A source code analysis framework"
  url="http://clang-analyzer.llvm.org/"
  license=('custom:University of Illinois/NCSA Open Source License')
  depends=("clang=$pkgver" "python")
}

$ ./pkgparse PKGBUILD
pkgname=[llvm, clang, clang-analyzer]
pkgver=2.6
pkgrel=3.0
source=[http://llvm.org/releases/2.6/llvm clang clang-analyzer-2.6.tar.gz, http://llvm.org/releases/2.6/clang-2.6.tar.gz]
arch=[i686, x86_64]
makedepends=[gcc-libs, python]

Split package: llvm
    pkgdesc=Low Level Virtual Machine
    url=http://llvm.org/
    licenses=[custom:University of Illinois/NCSA Open Source License]
    depends=[perl]
    makedepends=[gcc-libs, python]

Split package: clang
    pkgdesc=C language family frontend for LLVM
    url=http://clang.llvm.org/
    licenses=[custom:University of Illinois/NCSA Open Source License]
    depends=[gcc-libs]
    makedepends=[gcc-libs, python]

Split package: clang-analyzer
    pkgdesc=A source code analysis framework
    url=http://clang-analyzer.llvm.org/
    licenses=[custom:University of Illinois/NCSA Open Source License]
    depends=[clang=2.6, python]
    makedepends=[gcc-libs, python]

Note: Empty values were removed from the output. The input PKGBUILD was also modified to remove the actual body of functions (the parser doesn't recognize some of the grammar yet, but it would ignore it anyway).

As you can see the parser works quite well, although there are still some bugs (for instance $pkgname expanding to the full array - I'll have to look into what makepkg does to work around that. Atm it complies to the standard).

Last edited by Xilon (2009-11-14 11:09:52)

djszapi

Member

From: Cambridge, United Kingdom

Registered: 2009-06-14

Posts: 1,439

Website

Re: The AUR webapp needs our help!

AUR project was established on BERLIOS with the initial release of the older works:
http://git.berlios.de/cgi-bin/gitweb.cg … ;a=summary

You can join to the mailing list too, if you're interested in this new generation AUR:
https://lists.berlios.de/mailman/listinfo/aur2-dev

At this momment, Sebastian and me are the Project Admins there (if Sebastian will be able to activate his account ) Any patch is welcome. Maybe it will be officially supported, but  lots of work/contribution is needed.

Sebastian and me could get  apart from the Berlios Project a good remote database(mysql) + django supported place across ssh connections to test/debug the development. Some modifications happened in the AUR2 wikipage too.

Last edited by djszapi (2009-11-14 19:15:04)

---

http://lpapp.blogspot.com

SpeedVin

Member

From: Poland

Registered: 2009-04-29

Posts: 955

Re: The AUR webapp needs our help!

Hello is there place for me in project I got some experience in:
Python/Django , XHTML/HTML , CSS , SHELL
Thanks

---

Shell Scripter | C/C++/Python/Java Coder | ZSH