jshon - parsing json in the shell

keenerd · 2011-02-15 21:52:50

In my never ending quest to make the AUR be easy to use from the command line, the JSON RPC has always been a stumbling block. Here is what I am talking about:

https://aur.archlinux.org/rpc.php?type= … =cower-git
http://aur3.org/rpc/jshon

Parsing this information in the shell sucks. Awk/grep/sed were not made for this sort of thing. Jshon is. Pipe json though stdin, and get info on standard out.

More details on the homepage: http://kmkeen.com/jshon/ It is also in [community].

Summary of options:

-t (type) returns string, object, array, number, bool, null
-l (length) returns an integer. Only works on string, object, array.
-k (keys) returns a newline separated list of keys. Only works on object.
-e index (extract) returns json value at "index". Only works on object, array.
-s value (string) creates a json encoded string.
-n value (nonstring) creates a json encoded number or empty object/array.
-u (unstring) returns a decoded string. Only works on simple types.
-a (across) instead of diving deeper, commands are applied across the breadth of an object or list.
-p (pop) removes the last manipulation
-d index (delete) removes the value at index
-i index (insert) is complicated. It is the reverse of extract. Extract saves a copy of the json on a stack. Insert pops json from the stack, and inserts that bit of json into the new top of the stack. Use extract to dive into the json tree, delete/string/nonstring to change things, and insert to push the changes back up the tree.

Looking for people to test it out.

Credit and thanks goes to Falconindy for kicking the code into compilable shape.

Last edited by keenerd (2011-12-22 13:17:43)

keenerd · 2011-02-15 23:57:15

An example. In Aurphan, there is an alarming bit of awk:

curl -s "http://aur.archlinux.org/rpc.php?type=info&arg=cower" | awk '
    BEGIN       { RS=","; FS=":" }
    /type/      { if ($2 ~ /error/) exit }
    /ID/        { gsub(/\"/, "", $3); print $3 }
    /NumVotes/  { gsub(/\"/, "", $2); print $2 }
    /OutOfDate/ { gsub(/\"|}/, "", $2); print $2; exit }'

which can be replaced with

rpc=$(curl -s "http://aur.archlinux.org/rpc.php?type=info&arg=cower")
if [[ $(echo "$rpc" | jshon -e info -u) == "error" ]]; then return; fi
echo "$rpc" | jshon -e results -e ID -u
echo "$rpc" | jshon -e results -e NumVotes -u
echo "$rpc" | jshon -e results -e OutOfDate -u

or if you want to be a ninja

rpc=$(curl -s "http://aur.archlinux.org/rpc.php?type=info&arg=cower")
if [[ $(jshon -einfo -u <<< "$rpc") == "error" ]]; then return; fi
jshon -eresults -eID -uiID -eNumVotes -uiNV -eOutOfDate -u <<< "$rpc"

The jshon oneliner is also 8 times faster than the awk oneliner.

Last edited by keenerd (2011-02-16 01:43:51)

cactus · 2011-02-16 01:16:28

keenerd. If you are interested in networking code (having jshon do the fetching too), feel free to re-use any code from cronkite (MIT licensed). Currently it uses cJSON, but I have been thinking of moving to yajl but haven't yet -- no outside interest in the project makes me lazy.

Last edited by cactus (2011-02-16 01:19:01)

keenerd · 2011-02-16 14:09:05

Thanks. Though for now I'd rather have it do one thing well and not include another dep. Falconindy has been doing some fun work with a pure bash downloader, who needs libcurl? :-)

cactus · 2011-02-16 19:48:31

If you want to support gzip compression, proxies, or https, then libcurl is nice.

keenerd · 2011-05-09 07:08:04

Now in community.

amazergling · 2011-05-30 06:19:08

This console utility is interesting.I tested it with my json format file,it works fine.But I have some questions.

1.How can I traverse an array like?

for example,i have a json like this:
PHP:
$array = array(
'children'=>array(
'0'=> array( 'name'=>'son1', 'gender'=>'boy'),
'1'=> array( 'name'=>'son2', 'gender'=>'boy'),
'2'=> array( 'name'=>'daughter1', 'gender'=>'girl')
)
);

JSON FROMAT:
{"children":[{"name":"son1","gender":"boy"},{"name":"son2","gender":"boy"},{"name":"daughter1","gender":"girl"}]}

how can I fetch all children's name and gender~?

2.Is there some possible that users can access tree node via path like "a/b/c" in the future? XMLStarlet has such feature so convenience.

Last edited by amazergling (2011-05-30 06:21:15)

keenerd · 2011-05-30 11:15:24

1) Loops! (Please ignore useless cat and other bad habits.)

number=$(cat kids.json | jshon -e children -l)
for i in $( seq 0 $(( $number - 1 )) ); do
    cat kids.json | jshon -e children -e $i -e name -u -i n -e gender -u
done

I have some ideas for streamlining this, (built in support for loops) but still not sure how to implement it.

2) Not really. There is no standard "xpath" for json. Maybe key1.key2.key3 but that is a javascript thing.

keenerd · 2011-07-20 20:04:00

Looping has been completely overhauled. Here is an example for searching the AUR:

curl -s 'https://aur.archlinux.org/rpc.php?type=search&arg=test' | \
jshon -e results -a -e Name -u

Last edited by keenerd (2011-07-20 20:04:22)

UncleNinja · 2011-12-14 17:08:08

This tool is great!

Is there a way to move upward without undoing changes?

keenerd · 2011-12-15 03:02:54

Yes. If you are using the current version in the repos, you can insert (-i) the changes back up the stack. If you are using the git version you can also pop (-p) the lower levels as the changes affect the entire stack.

Come to think of it, I really should release the git version tonight. It runs about twice as fast when looping over large files.

Last edited by keenerd (2011-12-15 03:03:21)

Cloudef · 2011-12-15 05:34:53

This looks really good, it might replace jsawk for me. Thank you

sdellysse · 2011-12-20 06:27:44

I'm working on converting rc.conf into rc.conf.json, and I wanted to be able to keep comments in the file, so I whipped this up quickly:

strip-comments () {
    sed  's/\s*\/\/.*$//'
}

It (should) remove all comments in the style of

// Lorem Ipsum

Invoked as such:

HOSTNAME=$(cat /etc/rc.conf.json | strip-comments | jshon -e hostname -u)

Last edited by sdellysse (2011-12-20 06:27:55)

keenerd · 2011-12-22 13:08:26

New release! Additional features include a faster, more efficient use of jansson and a continue-through-errors mode that Karol requested.

karol · 2011-12-22 19:15:28

Look what you did, keenerd: now I need a bigger tree, cause this great app of yours won't fit under the old one.
A big big big thank you :-)

collards · 2012-04-23 18:21:11

Hi there,

I've tried several different options but jshon is really awesome so far, so thanks for that.

That being said, I am really running into trouble trying to use jshon for a big json file (around 150k lines).

The program isn't throwing any errors that I can see in standard output. It just doesn't ever complete. Any ideas or recommendations?

I have no problem "chunking" the file, but as you all probably know, json isn't great for chunking with something like linux cut.

Thanks

keenerd · 2012-04-24 03:27:39

Can you share a few lines of the file? (Something that I could repeat a few thousand times for testing.) I just tried a simple 150k line input and there were no problems, but maybe your input is more complicated.

markuskress · 2016-11-15 12:21:05

Hello keenerd,

old thread, nice tool. I have a question about inserting new array elements. Using -i 0 adds a new element at the first position, existing elements shifted behind. How is it possible to add a new array element at the end?

Example:
jshon -Q -n {} -n [] -i array > example.json
cat example.json | jshon -e array -s "element1" -i 0 -p > example2.json
cat example2.json | jshon -e array -s "element2" -i 0 -p > example3.json
cat example3.json | jshon -S

{
"array": [
"element2",
"element1"
]
}

Thanks

jjacky · 2016-11-16 21:40:58

Use the string "append" instead of an integer as index:

$ jshon -Q -n {} -n '[]' -i array | jshon -e array -s "element1" -i 0 -p | jshon -e array -s "element2" -i append -p | jshon -S
{
 "array": [
  "element1",
  "element2"
 ]
}

ychaouche · 2022-01-30 14:34:07

Hey all, nice tool. How do I go about getting the value of multiple keys instead of a single one ?
For example

jshon -e (version + id + name) < data.json

Trilby · 2022-01-30 15:16:05

FYI, this is an 11 year old thread for software than hasn't been updated in 8 years and even the AUR package was orphaned and last maintained several years ago. I'd not be surprised if jshon fell to the wayside as `jq` will do all of this very well. You'll most likely want to use jq instead.

Arch Linux

#1 2011-02-15 21:52:50

jshon - parsing json in the shell

#2 2011-02-15 23:57:15

Re: jshon - parsing json in the shell

#3 2011-02-16 01:16:28

Re: jshon - parsing json in the shell

#4 2011-02-16 14:09:05

Re: jshon - parsing json in the shell

#5 2011-02-16 19:48:31

Re: jshon - parsing json in the shell

#6 2011-05-09 07:08:04

Re: jshon - parsing json in the shell

#7 2011-05-30 06:19:08

Re: jshon - parsing json in the shell

#8 2011-05-30 11:15:24

Re: jshon - parsing json in the shell

#9 2011-07-20 20:04:00

Re: jshon - parsing json in the shell

#10 2011-12-14 17:08:08

Re: jshon - parsing json in the shell

#11 2011-12-15 03:02:54

Re: jshon - parsing json in the shell

#12 2011-12-15 05:34:53

Re: jshon - parsing json in the shell

#13 2011-12-20 06:27:44

Re: jshon - parsing json in the shell

#14 2011-12-22 13:08:26

Re: jshon - parsing json in the shell

#15 2011-12-22 19:15:28

Re: jshon - parsing json in the shell

#16 2012-04-23 18:21:11

Re: jshon - parsing json in the shell

#17 2012-04-24 03:27:39

Re: jshon - parsing json in the shell

#18 2016-11-15 12:21:05

Re: jshon - parsing json in the shell

#19 2016-11-16 21:40:58

Re: jshon - parsing json in the shell

#20 2022-01-30 14:34:07

Re: jshon - parsing json in the shell

#21 2022-01-30 15:16:05

Re: jshon - parsing json in the shell

Board footer