You are not logged in.

#1 2011-02-15 21:52:50

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

jshon - parsing json in the shell

In my never ending quest to make the AUR be easy to use from the command line, the JSON RPC has always been a stumbling block.  Here is what I am talking about:

https://aur.archlinux.org/rpc.php?type= … =cower-git
http://aur3.org/rpc/jshon

Parsing this information in the shell sucks.  Awk/grep/sed were not made for this sort of thing.  Jshon is.  Pipe json though stdin, and get info on standard out.

More details on the homepage:  http://kmkeen.com/jshon/  It is also in [community].

Summary of options:

-t (type) returns string, object, array, number, bool, null
-l (length) returns an integer.  Only works on string, object, array.
-k (keys) returns a newline separated list of keys.  Only works on object.
-e index (extract) returns json value at "index".  Only works on object, array.
-s value (string) creates a json encoded string.
-n value (nonstring) creates a json encoded number or empty object/array.
-u (unstring) returns a decoded string.  Only works on simple types.
-a (across) instead of diving deeper, commands are applied across the breadth of an object or list.
-p (pop) removes the last manipulation
-d index (delete) removes the value at index
-i index (insert) is complicated.  It is the reverse of extract.  Extract saves a copy of the json on a stack.  Insert pops json from the stack, and inserts that bit of json into the new top of the stack.  Use extract to dive into the json tree, delete/string/nonstring to change things, and insert to push the changes back up the tree.

Looking for people to test it out.

Credit and thanks goes to Falconindy for kicking the code into compilable shape.

Last edited by keenerd (2011-12-22 13:17:43)

Offline

#2 2011-02-15 23:57:15

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

An example.  In Aurphan, there is an alarming bit of awk:

curl -s "http://aur.archlinux.org/rpc.php?type=info&arg=cower" | awk '
    BEGIN       { RS=","; FS=":" }
    /type/      { if ($2 ~ /error/) exit }
    /ID/        { gsub(/\"/, "", $3); print $3 }
    /NumVotes/  { gsub(/\"/, "", $2); print $2 }
    /OutOfDate/ { gsub(/\"|}/, "", $2); print $2; exit }'

which can be replaced with

rpc=$(curl -s "http://aur.archlinux.org/rpc.php?type=info&arg=cower")
if [[ $(echo "$rpc" | jshon -e info -u) == "error" ]]; then return; fi
echo "$rpc" | jshon -e results -e ID -u
echo "$rpc" | jshon -e results -e NumVotes -u
echo "$rpc" | jshon -e results -e OutOfDate -u

or if you want to be a ninja

rpc=$(curl -s "http://aur.archlinux.org/rpc.php?type=info&arg=cower")
if [[ $(jshon -einfo -u <<< "$rpc") == "error" ]]; then return; fi
jshon -eresults -eID -uiID -eNumVotes -uiNV -eOutOfDate -u <<< "$rpc"

The jshon oneliner is also 8 times faster than the awk oneliner.

Last edited by keenerd (2011-02-16 01:43:51)

Offline

#3 2011-02-16 01:16:28

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,615
Website

Re: jshon - parsing json in the shell

keenerd. If you are interested in networking code (having jshon do the fetching too), feel free to re-use any code from cronkite (MIT licensed). Currently it uses cJSON, but I have been thinking of moving to yajl but haven't yet -- no outside interest in the project makes me lazy.

Last edited by cactus (2011-02-16 01:19:01)


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#4 2011-02-16 14:09:05

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

Thanks.  Though for now I'd rather have it do one thing well and not include another dep.  Falconindy has been doing some fun work with a pure bash downloader, who needs libcurl? :-)

Offline

#5 2011-02-16 19:48:31

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,615
Website

Re: jshon - parsing json in the shell

If you want to support gzip compression, proxies, or https, then libcurl is nice.
smile


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#6 2011-05-09 07:08:04

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

Now in community.

Offline

#7 2011-05-30 06:19:08

amazergling
Member
Registered: 2011-05-30
Posts: 1

Re: jshon - parsing json in the shell

This console utility is interesting.I tested it with my json format file,it works fine.But I have some questions.

1.How can I traverse an array like?

for example,i have a json like this:
PHP:
  $array = array(
    'children'=>array(
        '0'=> array( 'name'=>'son1', 'gender'=>'boy'),
        '1'=> array( 'name'=>'son2', 'gender'=>'boy'),
        '2'=> array( 'name'=>'daughter1', 'gender'=>'girl')
    )
  );

JSON FROMAT:
  {"children":[{"name":"son1","gender":"boy"},{"name":"son2","gender":"boy"},{"name":"daughter1","gender":"girl"}]}

how can I fetch all children's name and gender~?



2.Is there some possible that users can access tree node via path like "a/b/c" in the future? XMLStarlet has such feature so convenience.

Last edited by amazergling (2011-05-30 06:21:15)

Offline

#8 2011-05-30 11:15:24

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

1) Loops!  (Please ignore useless cat and other bad habits.)

number=$(cat kids.json | jshon -e children -l)
for i in $( seq 0 $(( $number - 1 )) ); do
    cat kids.json | jshon -e children -e $i -e name -u -i n -e gender -u
done

I have some ideas for streamlining this,  (built in support for loops) but still not sure how to implement it.

2) Not really.  There is no standard "xpath" for json.  Maybe key1.key2.key3 but that is a javascript thing.

Offline

#9 2011-07-20 20:04:00

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

Looping has been completely overhauled.  Here is an example for searching the AUR:

curl -s 'https://aur.archlinux.org/rpc.php?type=search&arg=test' | \
jshon -e results -a -e Name -u

Last edited by keenerd (2011-07-20 20:04:22)

Offline

#10 2011-12-14 17:08:08

UncleNinja
Member
From: Georgia, USA
Registered: 2011-07-09
Posts: 10
Website

Re: jshon - parsing json in the shell

This tool is great!

Is there a way to move upward without undoing changes?

Offline

#11 2011-12-15 03:02:54

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

Yes.  If you are using the current version in the repos, you can insert (-i) the changes back up the stack.  If you are using the git version you can also pop (-p) the lower levels as the changes affect the entire stack.

Come to think of it, I really should release the git version tonight.  It runs about twice as fast when looping over large files.

Last edited by keenerd (2011-12-15 03:03:21)

Offline

#12 2011-12-15 05:34:53

Cloudef
Member
Registered: 2010-10-12
Posts: 636

Re: jshon - parsing json in the shell

This looks really good, it might replace jsawk for me. Thank you

Offline

#13 2011-12-20 06:27:44

sdellysse
Member
Registered: 2009-12-15
Posts: 26

Re: jshon - parsing json in the shell

I'm working on converting rc.conf into rc.conf.json, and I wanted to be able to keep comments in the file, so I whipped this up quickly:

strip-comments () {
    sed  's/\s*\/\/.*$//'
}

It (should) remove all comments in the style of

// Lorem Ipsum

Invoked as such:

HOSTNAME=$(cat /etc/rc.conf.json | strip-comments | jshon -e hostname -u)

Last edited by sdellysse (2011-12-20 06:27:55)

Offline

#14 2011-12-22 13:08:26

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

New release!  Additional features include a faster, more efficient use of jansson and a continue-through-errors mode that Karol requested.

Offline

#15 2011-12-22 19:15:28

karol
Archivist
Registered: 2009-05-06
Posts: 25,428

Re: jshon - parsing json in the shell

Look what you did, keenerd: now I need a bigger tree, cause this great app of yours won't fit under the old one.
A big big big thank you :-)

Offline

#16 2012-04-23 18:21:11

collards
Member
Registered: 2010-10-08
Posts: 5

Re: jshon - parsing json in the shell

Hi there,

I've tried several different options but jshon is really awesome so far, so thanks for that.

That being said, I am really running into trouble trying to use jshon for a big json file (around 150k lines). 

The program isn't throwing any errors that I can see in standard output.  It just doesn't ever complete.  Any ideas or recommendations?

I have no problem "chunking" the file, but as you all probably know, json isn't great for chunking with something like linux cut.

Thanks

Offline

#17 2012-04-24 03:27:39

keenerd
Trusted User (TU)
Registered: 2007-02-22
Posts: 647
Website

Re: jshon - parsing json in the shell

Can you share a few lines of the file?  (Something that I could repeat a few thousand times for testing.)  I just tried a simple 150k line input and there were no problems, but maybe your input is more complicated.

Offline

Board footer

Powered by FluxBB