jq - can't append objects to array in while loop

Question

I've got this service-accounts-list.txt file with service accounts:

serviceAccount:[email protected]
serviceAccount:[email protected]
serviceAccount:[email protected]
serviceAccount:[email protected]
serviceAccount:[email protected]

and this results.json file which I'd like to append data to:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    }
  ]
}

Expected result: I'd like to add an object for each line in service-accounts-list.txt, like so:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    }
  ]
}

My bash script so far looks like this:

while read p;
do
  cat result.json | jq --arg email "$p" '.access += [{"role": "WRITER", "userByEmail": $email}]' > result.json
done < service-accounts-list.txt

The resulting result.json file is actually empty. If I redirect the output to for example > result2.json it correctly adds the last service account, like so:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    }
  ]
}

so it seems the jq syntax is correct. I've tried adding the --unbuffered flag to no avail.

What am I missing?

Answer 1

Maybe the simplest efficient solution would be:

jq -nR --argfile result result.json '
  $result 
  | .access += [{role: "WRITER", userByEmail: inputs}] 
' service-accounts-list.txt

Notice -- no slurping, no splitting, no shell scripting. If the use of --argfile bothers you, feel free to use --slurpfile but "unslurp" $result by writing $result[0].

Answer 2

While obvious problem is that you are reading and redirecting to same file which you cannot do in bash and this answer has many useful explanations and work around for that.

But a better solution would be to totally avoid invoking jq multiple times in a loop and get it done in a single execution of jq.

# array to hold new values in json format
narr=()

# loop through input file and append json formatted values to array
while read -r line; do
    narr+=('{"role": "WRITER", "userByEmail": "'$line'"}')
done < service-accounts-list.txt

# call jq only once using -n option
jq -n 'input | .access += [inputs]' results.json <(printf '%s\n' "${narr[@]}")

Output:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    }
  ]
}