Parse JSON data in variables using JQ

Question

I have some problems with my script which collects data from a JSON, stores them in variables which are then used for a CURL request. I need to build a CURL request for EACH JSON entry.

My problem is that I would like to pass parameters to the CURL request, one by one. I was thinking about a for-loop but this won't be actually the right workaround.

This because

ruleId=$($whitelist | jq -r '.[].ruleId')

gives:

10055
10098

This can not be interpreted correctly from CURL.

So the question is, how can I pass variables in a proper manner, in a sort of iteration, using JQ? Again, I need to do single calls using CURLs for each entry in the JSON file.

Code:

$!/bin/sh
#set -e

# Info to test this script
# 1. Start docker 
# 2. Run this command from terminal: docker run -u zap -p 8080:8080 -i owasp/zap2docker-stable zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true
# 2. Get the JSON from api (test for now)
# 3. bash filter.sh
# 4. Check for filter to be set with browser http://localhost:8080/UI/alertFilter/ => view global filter

curl -s 'https://api.npoint.io/c29e3a68be632f73fc22' > whitelist_tmp.json
whitelist="cat whitelist_tmp.json"
listlength=$(jq '. | length' $whitelist)


ruleId=$($whitelist | jq -r '.[].ruleId')
alert=$($whitelist | jq -r '.[].alertName')
newLevel=$($whitelist | jq -r '.[].newLevel')
url=$($whitelist | jq -r '.[].url | .[]')
urlIsRegex=$($whitelist | jq -r '.[].urlIsRegex')
enabled=$($whitelist | jq -r '.[].enabled')
parameter=$($whitelist | jq -r '.[].parameter')
evidence=$($whitelist | jq -r '.[].evidence')
echo "Setting Rule for: $ruleId"
echo "$(curl --data-urlencode "ruleId=$ruleId" --data-urlencode "newLevel=$newLevel" --data-urlencode "url=$url" --data-urlencode "urlIsRegex=$urlIsRegex" --data-urlencode "enabled=$enabled" --data-urlencode "parameter=$parameter" --data-urlencode "evidence=$evidence" "http://localhost:8090/JSON/alertFilter/action/addGlobalAlertFilter")"

Answer 1

You can use bash arrays to store your values:

ruleId=($($whitelist | jq -r '.[].ruleId'))
alert=($($whitelist | jq -r '.[].alertName'))
...

and then iterate over them. Example:

for (( i = 0; i < "${#ruleId[@]}"; i++ )); do
  id="${ruleId[i]}"
  al="${alert[i]}"
  ...
  echo "$(curl --data-urlencode "ruleId=$id" ...
done

This works if and only if the values returned by your commands are single words (no spaces in them) or they are properly quoted. If you have more complex values you cannot simply assign them to an array with array=($(command)). You would get more cells than values in your array.

Answer 2

Pulling the comments from your previous question:

whitelist="whitelist_tmp.json"
listlength=$(jq '. | length' "${whitelist}")

mapfile -t rule      < <(jq -r '.[].ruleId'      "${whitelist}")
mapfile -t alert     < <(jq -r '.[].cwalertName' "${whitelist}")
mapfile -t level     < <(jq -r '.[].newLevel'    "${whitelist}")
mapfile -t url       < <(jq -r '.[].url | .[]'   "${whitelist}")
mapfile -t regex     < <(jq -r '.[].urlIsRegex'  "${whitelist}")
mapfile -t parameter < <(jq -r '.[].parameter'   "${whitelist}")
mapfile -t evidence  < <(jq -r '.[].evidence'    "${whitelist}")

for ((i=0; i<${listlength}; i++))
do
    curl ... "${rule[$i]}" ... "${alert[$i]}" ...
done

The mapfile should maintain embedded white space in values returned by jq.