Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Split a JSON array into multiple files using command line tools

Tags:

json

bash

split

jq

Suppose we have a JSON array of length 5 and we want to split the array into multiple arrays of length 2 and save the grouped items into different files, using linux command line tools.

I tried it by using the jq and split tools (I am happy with any approach that can be executed from a bash script):

$ echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta_
$ tail -n +1 meta_*
==> meta_000 <==
{"key1":"value1"}
{"key2":"value2"}

==> meta_001 <==
{"key3":"value3"}
{"key4":"value4"}

==> meta_002 <==
{"key5":"value5"}

The previous command saves the items into the files correctly, but we need to convert them into a valid JSON array format. I tired with --filter option:

$ echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta2_ --filter='jq --slurp -c -M'
[{"key1":"value1"},{"key2":"value2"}]
[{"key3":"value3"},{"key4":"value4"}]
[{"key5":"value5"}]
$ tail -n +1 meta2_*
tail: cannot open 'meta2_*' for reading: No such file or directory

However, it displays the output on the screen but the results aren't persisted. I tried forwarding the output but I get an error:

echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta2_ --filter='jq --slurp -c -M > $FILE'
...
split: with FILE=meta2_000, exit 2 from command: jq --slurp -c -M > $FILE

Any hints or better approaches?

EDIT: I tried with double quotes @andlrc suggested:

$ echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta2_ --filter="jq --slurp -c -M > $FILE"
bash: -c: line 0: syntax error near unexpected token `newline'
bash: -c: line 0: `jq --slurp -c -M > '
split: with FILE=meta2_000, exit 1 from command: jq --slurp -c -M >
$ cat meta_000 | jq --slurp -c -M
[{"key1":"value1"},{"key2":"value2"}]
like image 329
Emer Avatar asked Nov 02 '16 17:11

Emer


1 Answers

It'll be easier to build out the arrays in the jq filter, then split to files per line. No additional filtering necessary.

range(0; length; 2) as $i | .[$i:$i+2]

produces:

[{"key1":"value1"},{"key2":"value2"}]
[{"key3":"value3"},{"key4":"value4"}]
[{"key5":"value5"}]

So putting it all together.

$ jq -cM --argjson sublen '2' 'range(0; length; $sublen) as $i | .[$i:$i+$sublen]' \
    input.json | split -l 1 -da 3 - meta2_
like image 97
Jeff Mercado Avatar answered Oct 15 '22 17:10

Jeff Mercado