Using jq to combine json files, getting file list length too long error

Question

Using jq to concat json files in a directory.

The directory contains a few hundred thousand files.

jq -s '.' *.json > output.json

returns an error that the file list is too long. Is there a way to write this that uses a method that will take in more files?

Answer 1

If jq -s . *.json > output.json produces "argument list too long"; you could fix it using zargs in zsh:

$ zargs *.json -- cat | jq -s . > output.json

That you could emulate using find as shown in @chepner's answer:

$ find -maxdepth 1 -name \*.json -exec cat {} + | jq -s . > output.json

"Data in jq is represented as streams of JSON values ... This is a cat-friendly format - you can just join two JSON streams together and get a valid JSON stream.":

$ echo '{"a":1}{"b":2}' | jq -s .
[
  {
    "a": 1
  },
  {
    "b": 2
  }
]

Answer 2

[EDITED to use find]

One obvious thing to consider would be to process one file at a time, and then "slurp" them:

$ while IFS= read -r f ; cat "$f" ; done <(find . -maxdepth 1 -name "*.json") | jq -s .

This however would presumably require a lot of memory. Thus the following may be closer to what you need:

#!/bin/bash
# "slurp" a bunch of files
# Requires a version of jq with 'inputs'.
echo "["
while read f
do
  jq -nr 'inputs | (., ",")' $f
done < <(find . -maxdepth 1 -name "*.json") | sed '$d'
echo "]"