jq with multiple inputs from different sources

Question

How can we mix different input sources when using jq ? For a specific usecase, I'd like to add some data from a file into a feed that was pipe in stdout.

$ echo '[{"a": 1}]' > /tmp/a1
$ echo '[{"a": 2}]' > /tmp/a2
$ jq --slurp '.[0] + .[1]' /tmp/a1 /tmp/a2
[
  {
    "a": 1
  },
  {
    "a": 2
  }
]
$ cat /tmp/a1 | jq --slurp '.[0] + .[1]' /tmp/a2  # Expecting the same result
[
  {
    "a": 2
  }
]

As you can see, the last command didn't interpret the piped data.

Right now, I'm forced to save the output from the first operation into a temporary file, so that I can do the jq merging operation, before sending it back to the network. Having a single stream would be much more efficient

Answer 1

I'd like to add some data from a file into a feed that was pipe in stdout.

There are various ways to do this, depending on the shell and also the version of jq you are using.

Assuming your jq supports the --argfile option, you might find that quite congenial:

cat /tmp/a1 | jq --argfile a2 /tmp/a2 '. + $a2'

Here is another variation that suggests some of the other possibilities:

jq -n --argfile a1 <(cat /tmp/a1) --argfile a2 <(cat /tmp/a2) '$a1 + $a2'

More interestingly:

(cat /tmp/a1 ; cat /tmp/a2) | jq '. + input' 

You might also wish to consider using the --slurpfile option instead of --argfile, but note that --slurpfile always "slurps" the file.

And finally an approach that should work for every version of jq:

jq -s '.[0] + .[1]' <(cat /tmp/a1) /tmp/a2

In general, though, it's best to avoid the -s option.

A note on slurping

If you compare the outputs produced by:

echo '1 2' |
  jq -s --debug-dump-disasm --debug-trace  '.[0], .[1]'

and

echo '1 2' |
  jq --debug-dump-disasm --debug-trace  '., input'

you'll notice the former has to PUSHK_UNDER to store the entire array [1,2], whereas the latter program just reads the two inputs separately.

In the first program, the memory for the array cannot be freed until after all the pointers into it have been processed, whereas in the second program, the memory for . can be freed after the first RET.

Answer 2

You could do this, where cat forwards its stdin followed by a2:

<GENERATE a1> | cat - /tmp/a2 | jq --slurp '.[0] + .[1]' 

Or this, which is a compound statement passing the results of two separate commands into a pipe:

{ <GENERATE a1> ; cat /tmp/a2; } | jq --slurp '.[0] + .[1]' 

Take care to have spaces beside the curly braces and to have a semi-colon before the final one.