Suppose we have a JSON array of length 5 and we want to split the array into multiple arrays of length 2 and save the grouped items into different files, using linux command line tools.
I tried it by using the jq
and split
tools (I am happy with any approach that can be executed from a bash script):
$ echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta_
$ tail -n +1 meta_*
==> meta_000 <==
{"key1":"value1"}
{"key2":"value2"}
==> meta_001 <==
{"key3":"value3"}
{"key4":"value4"}
==> meta_002 <==
{"key5":"value5"}
The previous command saves the items into the files correctly, but we need to convert them into a valid JSON array format. I tired with --filter
option:
$ echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta2_ --filter='jq --slurp -c -M'
[{"key1":"value1"},{"key2":"value2"}]
[{"key3":"value3"},{"key4":"value4"}]
[{"key5":"value5"}]
$ tail -n +1 meta2_*
tail: cannot open 'meta2_*' for reading: No such file or directory
However, it displays the output on the screen but the results aren't persisted. I tried forwarding the output but I get an error:
echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta2_ --filter='jq --slurp -c -M > $FILE'
...
split: with FILE=meta2_000, exit 2 from command: jq --slurp -c -M > $FILE
Any hints or better approaches?
EDIT: I tried with double quotes @andlrc suggested:
$ echo '[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"},{"key5":"value5"}]' | jq -c -M '.[]' | split -l 2 -d -a 3 - meta2_ --filter="jq --slurp -c -M > $FILE"
bash: -c: line 0: syntax error near unexpected token `newline'
bash: -c: line 0: `jq --slurp -c -M > '
split: with FILE=meta2_000, exit 1 from command: jq --slurp -c -M >
$ cat meta_000 | jq --slurp -c -M
[{"key1":"value1"},{"key2":"value2"}]
It'll be easier to build out the arrays in the jq filter, then split to files per line. No additional filtering necessary.
range(0; length; 2) as $i | .[$i:$i+2]
produces:
[{"key1":"value1"},{"key2":"value2"}]
[{"key3":"value3"},{"key4":"value4"}]
[{"key5":"value5"}]
So putting it all together.
$ jq -cM --argjson sublen '2' 'range(0; length; $sublen) as $i | .[$i:$i+$sublen]' \
input.json | split -l 1 -da 3 - meta2_
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With