What is the most efficient way of writing a JSON file with Bash?

Question

So I have to write a JSON file with bash script, and I know I can do something like echo 'something' >> $file to slowly build a file, but echo redirection instead of real file output seems kind of "hacky." If that is the best way, and not a hacky way at all, I am happy to use echo, but I was just wondering if there is a better way to output a file from bash script.

Answer 1

Efficiently generating output

echo is a built-in, not an external command, so it's not nearly as inefficient as you think. What is inefficient is putting >> filename on the end of each echo.

This is bad:

# EVIL!
echo "something" >file
echo "first line" >>file
echo "second line" >>file

This is good:

# NOT EVIL!
{
  echo "something" >&3
  printf '%s\n' "first line" "$second line" >&3
  # ... etc ...
} 3>file

...opens the output file only once, eliminating the major inefficiency.

To be clear: Calling echo, say, 20 times is considerably more efficient than calling cat once, since cat is an external process, not part of the shell. What's highly inefficient about running echo "foo" >>file 20 times is opening and closing the output file 20 times; it's not echo itself.

---

Correctly generating JSON

Don't use cat, echo, printf, or anything else of the sort. Instead, use a tool that understands JSON -- any other approach will lead to potentially incorrect (perhaps even exploitable via injection attacks) results.

For instance:

jq \
  --arg something "$some_value_here" \
  --arg another "$another_value" \
  '.["something"]=$something | .["another_value"]=$another' \
  <template.json >output.json

...will generate a JSON file, based on template.json, with something set to the value in the shell variable "$some_value_here" and another_value set to, well, a second value. Unlike naive approaches, this will correctly handle variable values which contain literal quotes or other characters which need to be escaped to be correctly represented.

---

An aside on echo

All the above having been said -- echo should be avoided in favor of printf (with an appropriate, static format string). Per the POSIX sh standard:

APPLICATION USAGE

It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.

The printf utility can be used portably to emulate any of the traditional behaviors of the echo utility as follows (assuming that IFS has its standard value or is unset):

[...]

New applications are encouraged to use printf instead of echo.

RATIONALE

The echo utility has not been made obsolescent because of its extremely widespread use in historical applications. Conforming applications that wish to do prompting without s or that could possibly be expecting to echo a -n, should use the printf utility derived from the Ninth Edition system.

As specified, echo writes its arguments in the simplest of ways. The two different historical versions of echo vary in fatally incompatible ways.

The BSD echo checks the first argument for the string -n which causes it to suppress the that would otherwise follow the final argument in the output.

The System V echo does not support any options, but allows escape sequences within its operands, as described for XSI implementations in the OPERANDS section.

The echo utility does not support Utility Syntax Guideline 10 because historical applications depend on echo to echo all of its arguments, except for the -n option in the BSD version.