cronkite -- pointless aur/json snarfle util/library

cactus · 2010-01-25 21:25:25

cronkite performs info and search queries against the aurjson interface, returning results in tab delimited format.
homepage: http://github.com/cactus/cronkite

usage example:

cronkite -search vim | awk -F\t '{print $1"-"$3}' | tail -n +2

Note: the tail command is to rip off the header that gets output first. I have thought of just leaving that off, and documenting the order of elements output in the manpage/help files.

As I said, marginally pointless. Possibly useful for people making bash script/utilities that don't want to parse json results and instead just want something easily awk'able.

Last edited by cactus (2010-03-07 09:15:59)

Xyne · 2010-01-25 21:40:19

Why did you name it cronkite? Am I missing some blatant reference?

/curious

cactus · 2010-01-25 21:49:54

Xyne wrote:

Why did you name it cronkite? Am I missing some blatant reference?
/curious

walter cronkite was an old school newsman.
cronkite gets you info.

I dunno. I always use weird names for projects.

Xyne · 2010-01-26 00:29:49

I recognized the name, but I didn't make the "gets you info" connection (I didn't think Walter's scope was so limited ).

Could you include the raw output of cronkite in the OP? I think it would give a better idea of what you can do with it.

res · 2010-01-26 01:19:41

Good call on not formatting the output too much.

http://aur.archlinux.org/packages.php?ID=33959

Let me know if you'd like to maintain it.

e: Besides removing the header, I'd also suggest allowing NULL to be a delimiter.

Last edited by res (2010-01-26 02:02:22)

cactus · 2010-01-26 05:30:29

removed the header, instead opting to just document the output order in the manpage and readme.

@res. Thanks for packaging it up! Feel free to maintain the package of it if you desire, because I don't want to.

Null delimiter, eh? (I assume you mean \0 and not the string NULL, or null as in.. a space)
That would make it difficult to split up with awk wouldn't it?
In what situations would you find it useful?

I suppose I could read an env variable to let the user specify the delimit character.

$ CRONKITE_DELIMITER=" _TACOS_ " ./src/cronkite -info vim-supercat
vim-supercat _TACOS_ 28490 _TACOS_ 1.0-1 _TACOS_ 4 _TACOS_ Syntax for supercat configuration files _TACOS_ http://www.vim.org/scripts/script.php?script_id=2676 _TACOS_ /packages/vim-supercat/vim-supercat.tar.gz _TACOS_ custom _TACOS_ 3 _TACOS_ 0

cactus · 2010-01-26 05:44:30

Because I was so enamored with the possibility of using the string "_TACOS_", I have implemented that feature.

cactus · 2010-02-28 03:46:12

In an attempt to ... who the hell knows what... I split off the core functionality into a library (libcronkite) and the cronkite cli util now just uses that library.

A C library for fetching json data from the aur. Definitely silly!

# search resulted in 5059 package results at time of query...
#

# cronkite uses libcronkite, which uses libcurl and does --compress
time cronkite -search the > /dev/null
real    0m2.239s
user    0m0.125s
sys     0m0.043s

time curl -s --compressed "http://aur.archlinux.org/rpc.php?type=search&arg=the" -o /dev/null
real    0m1.420s
user    0m0.030s
sys     0m0.018s

# no --compressed
time curl -s "http://aur.archlinux.org/rpc.php?type=search&arg=the" -o /dev/null
real    0m5.131s
user    0m0.025s
sys     0m0.077s

time wget -q "http://aur.archlinux.org/rpc.php?type=search&arg=the" -O /dev/null
real    0m5.894s
user    0m0.019s
sys     0m0.085s

Not too bad, considering cronkite is copying strings around and parsing json.
Still a bit silly though!

Last edited by cactus (2010-02-28 04:00:02)

cactus · 2010-03-07 09:15:17

Added a python interface to the c library.
Well two actually. I couldn't decide if ctypes was better, or if a standard cmodule style interface was better. So I just did both to see.

cactus · 2010-03-10 05:29:43

Was playing around with the python modules I wrote, and combining it with slurpy as a test.
I "ported" slurpy to use libcronkite-python, and here are the results.

libcronkite-python (cmodule version)
real    0m0.465s
user    0m0.364s
sys     0m0.064s

libcronkite-python (ctypes version)
real    0m0.475s
user    0m0.385s
sys     0m0.064s

python only (regular slurpy)
real    0m1.587s
user    0m1.496s
sys     0m0.064s

The above was tested against a loopback netcat (just output an http header and a json dump) as a 'webserver' to reduce network time/latency in the test.
The jsondump result set size was 5113 packages (I think I did curl against the aur with a search string of 'the' or something).

Still have a few things to fix: unicode, an unhandled case where server returns an http 500 level error.

Goofy tidbit of the day, the aur rpc api doesn't seem to like unicode query strings.
The aur web search seems to convernt ñ to n, and just search for the ascii string. The aur.rpc appears to just puke.
I should file a bug on that...

EDIT: Filed

Last edited by cactus (2010-03-10 05:45:58)

rson451 · 2010-03-10 13:21:09

cactus wrote:

Was playing around with the python modules I wrote, and combining it with slurpy as a test.
I "ported" slurpy to use libcronkite-python, and here are the results.

Very cool. I'll have to check out your python lib, I hadn't really looked at cronkite since it was just a standalone C program.

Goofy tidbit of the day, the aur rpc api doesn't seem to like unicode query strings.
The aur web search seems to convernt ñ to n, and just search for the ascii string. The aur.rpc appears to just puke.
I should file a bug on that...
EDIT: Filed

Nice find. That's a test case I've never even thought of.

Last edited by rson451 (2010-03-10 13:22:08)

cactus · 2010-03-10 22:32:14

hmm. maybe i should split out the codebase in github into libcronkite then, and separate the standalone utility.
I imagine that the library is far more useful than the demo cli frontend at this point.

Good call!

cactus · 2010-03-15 06:07:43

Did a fairly major refactor. The library is now the first class citizen, and the cli example is just that..an example of how to use the c library.
Other examples include a python cmodule, and a python ctypes based module.
I haven't decided which I like more..so I have kept them both around. The ctypes module is cleaner, but the cmodule is every so slightly faster (not in the library, but in calling the library) -- this probably matters even less since the library entry point is generally only called once and all the data is returned. *shrug*

cactus · 2010-03-29 03:53:15

Added a lua c module.
Refactored, and made the build system a bit cleaner.

Arch Linux

#1 2010-01-25 21:25:25

cronkite -- pointless aur/json snarfle util/library

#2 2010-01-25 21:40:19

Re: cronkite -- pointless aur/json snarfle util/library

#3 2010-01-25 21:49:54

Re: cronkite -- pointless aur/json snarfle util/library

#4 2010-01-26 00:29:49

Re: cronkite -- pointless aur/json snarfle util/library

#5 2010-01-26 01:19:41

Re: cronkite -- pointless aur/json snarfle util/library

#6 2010-01-26 05:30:29

Re: cronkite -- pointless aur/json snarfle util/library

#7 2010-01-26 05:44:30

Re: cronkite -- pointless aur/json snarfle util/library

#8 2010-02-28 03:46:12

Re: cronkite -- pointless aur/json snarfle util/library

#9 2010-03-07 09:15:17

Re: cronkite -- pointless aur/json snarfle util/library

#10 2010-03-10 05:29:43

Re: cronkite -- pointless aur/json snarfle util/library

#11 2010-03-10 13:21:09

Re: cronkite -- pointless aur/json snarfle util/library

#12 2010-03-10 22:32:14

Re: cronkite -- pointless aur/json snarfle util/library

#13 2010-03-15 06:07:43

Re: cronkite -- pointless aur/json snarfle util/library

#14 2010-03-29 03:53:15

Re: cronkite -- pointless aur/json snarfle util/library

Board footer