How to loop over jq unique array in bash?

Question

I'm trying to loop over unique names and commit messages from a github json object. However when there are spaces in the arrays bash will treat them as individual array items

#!/usr/bin/env bash

commits='[
  {
    "author": {
      "email": "[email protected]",
      "name": "Chris",
      "username": "chris"
    },
    "committer": {
      "email": "[email protected]",
      "name": "Chris",
      "username": "chris"
    },
    "message": "commit message 1"
  },
  {
    "author": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "committer": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "message": "commit message 2"
  },
    {
    "author": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "committer": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "message": "commit message 3"
  }
]'

authors=$( jq -rc '[.[].author.name] | unique | @sh' <<<"${commits}" )
echo "authors: $authors"

# this works
for author in $authors
do
  echo "author: $author"
done

echo "------------"

# this does not
messages=$( jq -rc '[.[].message] | unique | @sh' <<<"${commits}" )
echo "messages: $messages"

for message in $messages
do
  echo "message: $message"
done

Which outputs

authors: 'Chris' 'John'
author: 'Chris'
author: 'John'
------------
messages: 'commit message 1' 'commit message 2' 'commit message 3'
message: 'commit
message: message
message: 1'
message: 'commit
message: message
message: 2'
message: 'commit
message: message
message: 3'

While I expect:

authors: 'Chris' 'John'
author: 'Chris'
author: 'John'
------------
messages: 'commit message 1' 'commit message 2' 'commit message 3'
message: 'commit message 1'
message: 'commit message 2'
message: 'commit message 3'

Answer 1

Use readarray (Bash 4+) to map null delimited output from jq:

#!/usr/bin/env bash

commits='[
  {
    "author": {
      "email": "[email protected]",
      "name": "Chris",
      "username": "chris"
    },
    "committer": {
      "email": "[email protected]",
      "name": "Chris",
      "username": "chris"
    },
    "message": "commit message 1"
  },
  {
    "author": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "committer": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "message": "commit message 2"
  },
    {
    "author": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "committer": {
      "email": "[email protected]",
      "name": "John",
      "username": "jdoe"
    },
    "message": "commit message 3"
  }
]'

readarray -d '' authors < <(jq -j '.[].author.name + "\u0000"' <<<"${commits}")

for author in "${authors[@]}"
do
  echo "author: $author"
done

echo "------------"

readarray -d '' messages < <(jq -j '.[].message + "\u0000"' <<<"${commits}")

for message in "${messages[@]}"
do
  echo "message: $message"
done

Alternatively, if you have an older Bash version without readarray or mapfile you may separate the strings with the ASCII control character ETX (End of TeXt 03) and use read instead like this:

IFS=$'\03' read -d '' -ra authors < <(jq -j '.[].author.name + "\u0003"' <<<"${commits}")

IFS=$'\03' read -d '' -ra messages < <(jq -j '.[].message + "\u0003"' <<<"${commits}")

It is also possible to populate both arrays from a single jq call:

# Populates both arrays from a single jq call
{
  IFS=$'\03' read -r -d '' -a authors
  IFS=$'\03' read -r -d '' -a messages
} < <(jq -j '([.[].author.name] | unique | .[] + "\u0003"), "\u0000",  ([.[].message] | unique | .[] + "\u0003")' <<<"${commits}")

Explanation:

• [.[].author.name] | unique | .[] + "\u0003":
  Output an ETX (03) delimited list of unique author names.

• "\u0000": insert a null delimiter

• [.[].message] | unique | .[] + "\u0003":
  Output an ETX (03) delimited list of unique messages.

• Feeds the whole output of jq to a command group with two read commands.
  Each read will stop at the null delimiter or end of the stream.

{
  IFS=$'\03' read -r -d '' -a authors
  IFS=$'\03' read -r -d '' -a messages
}