AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

extofme · 2010-06-25 08:08:01

Welcome to AUR3.

available (pre-alpha)...
http://aur.archlinux.org/packages.php?ID=38359

QUICKSTART

1) install
# http://aur.archlinux.org/packages.php?ID=38359
2) run
# /usr/lib/aur-pyjs-git/aur serve -f
3) view
# http://127.0.0.1:8000/Aur.html

INTRODUCTION

This is a reimplementation of the AUR using a widget toolkit known as Pyjamas (http://pyjs.org). Client-side code (to run in web browsers) is written and maintained in python; when deploying to production, the python sources are compiled/translated to 100% pure javascript. The resulting javascript code can be ran on any major browser. When deployed, this is a 100% pure javascript application... thus, javascript is required, else you will see only a blank page.

WHY PYJS?

1) anyone that knows python can write clean, maintainable client-side code
2) eliminates the need to know the umpteen inconsistencies amongst browsers
3) via pyjamas-desktop, the app runs as 100% pure python, as a true desktop app, with no modifications
4) back-ends are JSON-RPC; allows back-ends to be written in any language, and enforces a clean separation

PROJECT STATUS

1) FRAMEWORK
[complete] basic URL dispatcher
[complete] load new content pages with cache support
[new] create generic URL-to-module dispatcher (like cherrypy/etc.)
2) SHELL
[complete] pixel perfect to other sections of the site
[complete] links/titles/copyright in place
3) BASIC SEARCH
[complete] search front-end
[incomplete] perform search front-end
[incomplete] perform search back-end
4) ADVANCED SEARCH
[complete] define toggles
[incomplete] define filters
[incomplete] define sorts
[incomplete] enable/enforce limits
[complete] search front-end
[incomplete] perform search front-end
[incomplete] perform search back-end
5) LANGUAGES
[incomplete] Language.py module methods
[incomplete] language JSON-RPC backend (preferred), or hardcode in Language.py
[incomplete] replace hardcoded text with calls to Language module
6) HOME
[complete] create page
[incomplete] introduction/disclaimer
[incomplete] recent updates front-end
[incomplete] recent updates back-end
[incomplete] statistics front-end
[incomplete] statistics back-end
7) BROWSE/SEARCH/MY PACKAGES
[complete] create page
[incomplete] paginating results front-end
[incomplete] paginating results back-end
8) VIEW PACKAGE
[incomplete] create page
[incomplete] package details front-end (name/link/desc/deps/files/etc.)
[incomplete] package details back-end
[incomplete] list comments front-end
[incomplete] list comments back-end
[incomplete] add comment front-end
[incomplete] add comment back-end
9) ACCOUNTS
[incomplete] create page
[incomplete] create/edit account front-end
[incomplete] create/edit account back-end
10) SUBMIT
[incomplete] create page
[incomplete] submit package front-end
[incomplete] submit package back-end

PYJAMAS-DESKTOP ONLY (as python)

1) INSTALL
[incomplete] add 'install' links (view/browse/search pages)
[incomplete] install status/details front-end (GUI)
[incomplete] python module "back-end" to download packages + dependencies and install

GET INVOLVED

1) set up a development environment + pyjs sandbox at ~/aur-pyjs...
# /usr/lib/aur-pyjs-git/aur sync ~/aur-pyjs
# cd ~/aur-pyjs
2) update pyjamas anytime by running...
# ./aur sync
3) generate the AUR...
# ./aur trans
4) view the AUR...
# ./aur serve -f
# http://127.0.0.1:8000/Aur.html
5) create a package based on remote git master...
# ./aur pkg
6) create a package based on your local git master...
# ./aur pkg -l
7) create a package based on your local git master AND install it...
# ./aur pkg -li

All of the commands support a 'help' and 'usage' parameter:

# ./aur help
Usage: aur-pyjs COMMAND [help|OPTION]...
Develop, translate, package, and serve AUR3

COMMANDs:
   pkg     build package from local/remote master; opt. install; opt. upload
   sync    bootstrap/update local/target devel environment
   trans   translate python sources to javascript for deployment
   serve   lastest local build at http://localhost:8000

# ./aur pkg help
Usage: aur-pyjs pkg [-l] [-i] [-u]
Package local/remote build and optionally install and/or upload to AUR legacy

Options:
   -l   favor local source over remote
   -i   install package locally
   -u   upload to AUR legacy after building

LAST WORDS

yes. I hope this to become the official AUR, and/or some other directions [1]
yes. running as a python desktop app, you will be able to install packages directly
yes. project is active; tentative completion date, ~October 2010
possibly. I'd like to make it capable of running independently, managing the local system
no. project is not a revived incarnation of other attempts

) advanced search will be restricted
) SOMEONE CAN START BUILDING JSON-RPC BACKENDS NOW
) PHP and MySQL s_ck. If I do it, the back-ends will be python ( + git/sqlite/couchdb)

) feedback appreciated
) to contribute, fork on github and send me a request.

C Anthony

[1] http://bbs.archlinux.org/viewtopic.php?id=90970

Last edited by extofme (2010-09-12 23:37:47)

extofme · 2010-06-28 10:32:02

) cleaned up the advanced search
) made the copyright configurable so i don't get sued :-), XXXXXX for now
) now properly depends on pyjamas package
) added pyjamas-git package
) cleaned up some bugs in PKGBUILD and src

im going to focus on making a search results page next. since rpc.php is JSON, if i feel ambitious, i might retrofit the current RPC so we can do real searches. good for testing, and at that point we would have a basic desktop app to the AUR.

C Anthony

krzd · 2010-06-29 14:50:09

Is this meant to go official as replace for the existing AUR?
If so, in which time can we expect it, 1 year, 5 years?

extofme · 2010-06-30 01:41:13

krzd wrote:

Is this meant to go official as replace for the existing AUR?
If so, in which time can we expect it, 1 year, 5 years?

1-5 yrs? not much faith in the AUR ever being replaced around here, is there?

i answered you in the first edited post. short answers:

1) that's the goal, if the powers that be are onboard
2) 2 months-ish. tops-ish

-----------------------

updates:

) edited first post
) added 'install' script
) moved PKGBUILD into distribution
) bug fixes all over the place

C Anthony

flamelab · 2010-06-30 02:15:55

Have you informed the aur-general mailing list so that TU's and devs can see it ?

extofme · 2010-06-30 02:40:00

flamelab wrote:

Have you informed the aur-general mailing list so that TU's and devs can see it ?

heh, not yet. i wanted to get a couple more things implemented, specifically the browse/search/my packages page, and maybe the home page.

but meh, feel free to start the party early :-)

C Anthony

extofme · 2010-06-30 22:25:47

) added README [first post, verbatim]
) more bugfixes to the service scripts

i have created a mailing list for this project; it receives a post from github whenever someone pushes commits:

aur-pyjs_at_extof.me

i have to add you manually (google apps group). lmk if interested

C Anthony

Xyne · 2010-06-30 23:12:35

Sorry for being too lazy right now to check myself, but would this require that Javascript be enabled to use the AUR or does Pyjamas gracefully fall back to plain HTML?

Regardless, it seems that you have a good vision for this project and I hope that your enthusiasm for it continues. I know that it can be somewhat discouraging when initially met with nay-saying and/or a seeming lack of interest.

extofme · 2010-06-30 23:53:40

Xyne wrote:

Sorry for being too lazy right now to check myself, but would this require that Javascript be enabled to use the AUR or does Pyjamas gracefully fall back to plain HTML?

heh, yeah i purposefully wasn't mentioning that at this point .

no, it is not able to fall back to plain old HTML. there was/is a project, pyjamas-server or something, where the backend/server would run the app as python (i think, might have been spidermonkey), compute the DOM, then spit the result to the client/browser as an HTML document. every time you did an "action" (clicked something/whatever), you would make a new request to the server, it would do the action for you, then spit back the result. i'm not sure completely how it worked, but the biggest problem was that in order to maintain a "state" from the clients perspective (simply retaining variables/etc.), a permanent/daemon process was required on the server. this daemon would essentially be doing the same thing a client browser would have done, had javascript been enabled, but the daemon is doing it for _every_ client that doesn't support JS. the other option was to serialize/deserialize the entire state after each request, but that's slow

i have some hope for this in the future, but ATM it's not here. although, it may be easy to "snapshot" points in a running pyjamas app, and save the DOM. this could then be served up, and could reuse the same CSS as the JS version; i believe this is sort of how the new test suite in pyjs works, but i'm not sure.

the other option would be to just create a super simple HTML only version, maybe try to reuse the CSS, i dunno. i've done manual web development for so long, that frankly i stopped caring about supporting 15 browsers and degrading gracefully to 10 different levels of functionality, it's just too much, and drives you mad wasting time. this is why i love pyjamas; it takes care of the top 5 browsers or so, and others usually work fine. so, i make a full functioning version, and a super crappy basic HTML version that does the bare minimum; if you don't like it, enable JS. with the advent of HTML5 and the JS API's, JS will start to become a hard requirement anyways i think.

that was my long winded way of saying "no".

Xyne wrote:

Regardless, it seems that you have a good vision for this project and I hope that your enthusiasm for it continues. I know that it can be somewhat discouraging when initially met with nay-saying and/or a seeming lack of interest.

thanks, and yeah it can be. haha i thought people would be all over the idea of a new AUR, esp. one capable of running as a desktop app/installing with one click. thats why i made it so easy to bootstrap a dev environment. but meh, i code because i enjoy it, so i keep doing it cuz it feels gooooood.

C Anthony

extofme · 2010-07-20 09:06:37

no big changes, but i've updated the bootstrap file to take advantage of new module loader code and wrote a super sexy awesome bootsplash + fade effect loading screen :-D

now depends on pyjamas-git

new UI widgets recently added upstream.

i think im going to rewrite what i have already to use PureMVC; this should simplify everything in the long[ish] run. i'm in the middle of a big move; more activity soon. completion date will probably slide a bit... deadlines make stuff not fun anyway :-)

C Anthony

Dieter@be · 2010-09-10 12:22:03

What's the status of this?
How is the package storage implemented? Knowing you, probably each package gets a git repo which people can fork so they can easily contribute to packages?
If so, have you talked about this with Loui? He's a git fan as well.

extofme · 2010-09-11 22:07:36

Dieter@be wrote:

What's the status of this?
How is the package storage implemented? Knowing you, probably each package gets a git repo which people can fork so they can easily contribute to packages?
If so, have you talked about this with Loui? He's a git fan as well.

i wish i could say that i finished it by Sep 1, but alas, i have not. i recently moved to a new state/city and im having difficulty finding a good development job.

i have implemented a small number of things that are not currently in git, but not much. i do have two very promising interviews this week... if that goes well, and i wrap up one other higher priority (finance critical) project, then ill be back to work on this full swing.

people promise/say stuff all the time, so i expect others to possibly think that here as well; however, this project and the underlying motivations for it are a _major_ goal for me, so i assure you and others that there will be a flurry of activity in the coming weeks.

i was hoping that others would install the AUR package, take a look, see the massive potential, and start tinkering; i put quite a bit of time into making a nice build and development system. i'm sure interest will rise once i've got it searching and installing from the current AUR, which it is very close to doing.

additionally, the work from my current project will be incorporated... a recursive content/layout generator that allows you to build any layout, and embed media into it (text/links/images/videos/etc.)... good stuff.

but yes, i will likely use git (heh, i must blabber about git a lot around here or something :-), or at least the data structure. one of the side projects of this initiative is to make Yet Another AUR Helper, but have it use this subsystem, thus providing a CLI or GUI interface to your system. i havn't talked to Loui (he is AUR maintainer right?), as i wasn't going to really annouce my work until more was done (but your distributed bugs post to the ML got me all fired up :-). However, i have read some posts from him on the AUR ML talking about a vision of the AUR that was decentralized... so he'd probably be all about this.

i'll see where i'm at next week, and if all looks good i plan to ramp this project up, and try to pull in some other python folk from the Arch community to help me get this thing rocking. will update.

C Anthony

Last edited by extofme (2010-09-11 22:09:42)

Dieter@be · 2010-09-12 09:16:28

extofme wrote:

i havn't talked to Loui (he is AUR maintainer right?), as i wasn't going to really annouce my work until more was done (but your distributed bugs post to the ML got me all fired up :-). However, i have read some posts from him on the AUR ML talking about a vision of the AUR that was decentralized... so he'd probably be all about this.

He is.

litemotiv · 2010-09-12 11:04:34

Interesting project, will be keeping an eye on this.

cesura · 2010-09-12 16:00:13

Very nice work. I can't wait to see where this project goes!

louipc · 2010-09-25 19:53:52

Nice.

I like that this looks easy to setup.
I like the idea of the client interface code running on the client side.
Does the implementation allow for non-javascript, non-browser clients yet?

In my mind a fully functional command line client/interface is essential
for the next generation implementation for the AUR.

extofme · 2010-09-25 21:01:07

louipc wrote:

Nice.
I like that this looks easy to setup.
I like the idea of the client interface code running on the client side.
Does the implementation allow for non-javascript, non-browser clients yet?
In my mind a fully functional command line client/interface is essential
for the next generation implementation for the AUR.

no not yet. i believe the direction i will head is the creation of a JSON-RPC daemon that runs on your local machine; this daemon will handle such things as:

) locating/communication with peers
) manipulating the system state (installing packages/etc.)
) manipulating and managing local repositories (public/private)
) accepting and authenticating incoming connections from clients

clients could be the pyjs GUI, an alternative GUI, a CLI, or anything else, so long as it speaks the JSON protocol (peer-to-peer communication at this point i'm thinking will be via SSH [paramiko], messaging [protocol buffers], and the git protocol [dulwich]). the cool thing about this approach, is you would be able to administer a system remotely using a GUI interface, or any other interface.

i think my job hunt is finally over, save one final... political... hurdle. i starting working on this again today actually, and am currently moving over to PureMVC, GUI updates, and updating the code significantly to make use of better practices i've acquired since first writing this.

ultimately, my real goal is to replace pacman itself with a different system that uses revision control concepts/libraries to manage the state, instead of the homebrew method pacman uses (which i'm not knocking or anything, this [system state management] is a point of much interest for me).

lastly, i am also preparing the code to be ran as a desktop app. pyjamas, in addition to python-JS translation, also has bindings to MSHTML/Trident, Webkit, and Gecko rendering engines (with GTK and QT possibly in the future). this feature allows an app to run as pure python, and use the full functionality of the python standard library, whilst also using the aforementioned engines to power a GUI.

interestingly, the MSHTML bindings means you could manage a Linux machine from Windows. another neat idea would be the ability to control mutliple machines at once, or choose a machine from a list to control, sort of like phpmyadmin; however, these features, in addition to replacing pacman, are outside the scope of AUR... so i will cross that bridge when it comes.

should have something interesting within a week or two.

C Anthony

Last edited by extofme (2010-09-25 21:04:03)

louipc · 2010-09-26 11:53:08

extofme wrote:

louipc wrote:
Does the implementation allow for non-javascript, non-browser clients yet?
In my mind a fully functional command line client/interface is essential
for the next generation implementation for the AUR.
no not yet. i believe the direction i will head is the creation of a JSON-RPC daemon that runs on your local machine; this daemon will handle such things as:
) locating/communication with peers
) manipulating the system state (installing packages/etc.)
) manipulating and managing local repositories (public/private)
) accepting and authenticating incoming connections from clients
clients could be the pyjs GUI, an alternative GUI, a CLI, or anything else, so long as it speaks the JSON protocol (peer-to-peer communication at this point i'm thinking will be via SSH [paramiko], messaging [protocol buffers], and the git protocol [dulwich]). the cool thing about this approach, is you would be able to administer a system remotely using a GUI interface, or any other interface.
i think my job hunt is finally over, save one final... political... hurdle. i starting working on this again today actually, and am currently moving over to PureMVC, GUI updates, and updating the code significantly to make use of better practices i've acquired since first writing this.
ultimately, my real goal is to replace pacman itself with a different system that uses revision control concepts/libraries to manage the state, instead of the homebrew method pacman uses (which i'm not knocking or anything, this [system state management] is a point of much interest for me).
lastly, i am also preparing the code to be ran as a desktop app. pyjamas, in addition to python-JS translation, also has bindings to MSHTML/Trident, Webkit, and Gecko rendering engines (with GTK and QT possibly in the future). this feature allows an app to run as pure python, and use the full functionality of the python standard library, whilst also using the aforementioned engines to power a GUI.
interestingly, the MSHTML bindings means you could manage a Linux machine from Windows. another neat idea would be the ability to control mutliple machines at once, or choose a machine from a list to control, sort of like phpmyadmin; however, these features, in addition to replacing pacman, are outside the scope of AUR... so i will cross that bridge when it comes.

Hehe I think I'm getting lost in the creative chaos. Well, I look forward to hearing about your progress. Cheers.

flamelab · 2010-09-26 13:00:09

extofme, your app looks soooo promising, I really want to see a working alpha/beta of it whenever you can, it could be awesome.-

extofme · 2010-09-26 20:22:08

louipc wrote:

Hehe I think I'm getting lost in the creative chaos. Well, I look forward to hearing about your progress. Cheers.

haha, i know i'm all over the place about this. i have been thinking about this long before i came to Arch even; the AUR was _easily_ the most attractive thing to me about Arch, and is the perfect platform to try some of these concepts.

flamelab wrote:

extofme, your app looks soooo promising, I really want to see a working alpha/beta of it whenever you can, it could be awesome.

:-) me too! i've been itching to work on this for weeks, but being broke and poor with a family, the job search has eaten all my time. alas, i landed a super-awesome-making-2X-what-i-was-before devel job last week, and providing one small... eh... hiccup... doesn't completely destroy it, i'll be on this full swing in my personal time.

doing my best to get a nice foundation to build on :-) working on it today; will update as soon as i can.

C Anthony

karol · 2011-01-16 14:54:29

Is this what AUR3 is going to become?

extofme · 2011-01-17 05:44:17

heh, nice catch :-)

thats the horizon target anyway. short term is a single node + interface providing at minimum the features of the current AUR. i had some failures/dataloss/changes/upheaval recently so i've been sinking time into my network, backup/recovery, and living space for some time now, but that's near finished ...

need to finish extensive release to mkinitcpio-btrfs, then back to this which is much more interesting. i really wanted this to be usable by last year, but i needed to secure employment after big move. luckily, everything is back on track now and settling out ...

... because i scored a position with TI after losing position the one for RP, and things have been really great ;-), universe works itself out if you put in enough energy i guess ...

so i will be hitting this again soon; though both branches on github "technically" work, they don't do anything interesting yet.

C Anthony

Dieter@be · 2011-01-21 22:44:44

extofme wrote:

need to finish extensive release to mkinitcpio-btrfs, then back to this which is much more interesting.

no it's not! both projects are awesome

extofme wrote:

... because i scored a position with TI after losing position the one for RP, and things have been really great ;-), universe works itself out if you put in enough energy i guess ...

congrats!

Xyne · 2011-01-21 23:25:50

extofme wrote:

... because i scored a position with TI after losing position the one for RP, and things have been really great ;-), universe works itself out if you put in enough energy i guess ...

Nice. Maybe you can bring this to their attention while you're there:

extofme · 2011-01-28 08:30:27

haha nice, thanks to both.

TI and RP were sadly just abbreviations i used to protect the innocent ... haha, it's not Texas Instruments

look in my recent history here and you'll soon discover the "error" of my ways and my infamous day of best in years to worst in longer ... there are plenty of ways to lose an excellent position before you start, after your hired/already selected, and without actually associating with anyone in between :-D

maybe someday we'll get past condemning each other for how one chooses to experience their life; bleh, i believe in us anyway.

C Anthony

Arch Linux

#1 2010-06-25 08:08:01

AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#2 2010-06-28 10:32:02

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#3 2010-06-29 14:50:09

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#4 2010-06-30 01:41:13

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#5 2010-06-30 02:15:55

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#6 2010-06-30 02:40:00

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#7 2010-06-30 22:25:47

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#8 2010-06-30 23:12:35

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#9 2010-06-30 23:53:40

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#10 2010-07-20 09:06:37

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#11 2010-09-10 12:22:03

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#12 2010-09-11 22:07:36

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#13 2010-09-12 09:16:28

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#14 2010-09-12 11:04:34

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#15 2010-09-12 16:00:13

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#16 2010-09-25 19:53:52

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#17 2010-09-25 21:01:07

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#18 2010-09-26 11:53:08

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#19 2010-09-26 13:00:09

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#20 2010-09-26 20:22:08

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#21 2011-01-16 14:54:29

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#22 2011-01-17 05:44:17

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#23 2011-01-21 22:44:44

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#24 2011-01-21 23:25:50

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

#25 2011-01-28 08:30:27

Re: AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC

Board footer