Bash parameter expansion, indirect reference, and backgrounding

Question

After struggling with this issue for several hours and searching here and failing to come up with a matching solution, it's time to ask:

In bash (4.3) I'm attempting to do a combination of the following:

• Create an array
• For loop through the values of the array with a command that isn't super fast (curl to a web server to get a value), so we background each loop to parallelize everything to speed it up.
• Set the names of the values in the array to variables assigned to values redirected to it from a command via "read"
• Background each loop and get their PID into a regular array, and associate each PID with the related array value in an associative array so I have key=value pairs of array value name to PID
• Use "wait" to wait for each PID to exit 0 or throw an error telling us which value name(s) in the array failed to exit with 0 by referencing the associative array
• I need to be able export all of the VAR names in the original array and their now-associated values (from the curl command results) because I'm sourcing this script from another bash script that will use the resulting exported VARs/values.

The reason I'm using "read" instead of just "export" with "export var=$(command)" or similar, is because when I background and get the PID to use "wait" with in the next for loop, I actually (incorrectly) get the PID of the "export" command which always exits 0, so I don't detect an error. When I use read with the redirect to set the value of the VAR (from name in the array) and background, it actually gets the PID of the command and I catch any errors in the next loop with the "wait" command.

So, basically, this mostly appears to work, except I realized the "read" command doesn't actually appear to be substituting the variable to the array name value properly in a way that the redirected command sends its output to that name in order to set the substituted VAR name to a value. Or, maybe the command is just entirely wrong so I'm not correctly redirecting the result of my command to a VAR name I'm attempting to set.

For what it's worth, when I run the curl | python command by hand (to pull the value and then parse the JSON output) it is definitely succeeding, so I know that's working, I just can't get the redirect to send the resulting output to the VAR name.

Here's a example of what I'm trying to do: In parent script:

# Source the child script that has the functions I need
source functions.sh

# Create the array
VALUES=(
VALUE_A
VALUE_B
VALUE_C
)

# Call the function sourced from the script above, which will use the above defined array
function_getvalues

In child (sourced) script:

function_getvalues()
{
  curl_pids=( )
  declare -A value_pids
  for value in "${VALUES[@]}"; do
    read ${value} < <(curl -f -s -X GET http://path/to/json/value | python3 -c "import sys, json; print(json.load(sys.stdin)['data']['value'])") & curl_pids+=( $! ) value_pids+=([$!]=${value})
  done
  for pid in "${curl_pids[@]}"; do
    wait "$pid" && echo "Successfully retrieved value ${value_pids[$pid]} from Webserver." || { echo "Something went wrong retrieving value ${value_pids[$pid]}, so we couldn't get the output data needed from Webserver. Exiting." ; exit 1 ; }
  done
}

Answer 1

The problem is that read, when run in the background, isn't connected to a standard in.^[details] Consider this simplified, working example with comment how to cripple it:

VALUES=( VALUE_A VALUE_B )
for value in "${VALUES[@]}"; do
    read ${value} < <(echo ${RANDOM}) # add "&" and it stops working
done
echo "VALUE_A=${VALUE_A}"
echo "VALUE_B=${VALUE_B}"

You might be able to do this with coproc, or using read -u with automatic file descriptor allocation, but really this is a job for temporary files:

tmpdir=$(mktemp -d)

VALUES=( VALUE_A VALUE_B )
for value in "${VALUES[@]}"; do
    (sleep 1; echo ${RANDOM} > "${tmpdir}"/"${value}") &
done
for value in "${VALUES[@]}"; do
    wait_file "${tmpdir}"/"${value}" && {
        read -r ${value} < "${tmpdir}"/"${value}";
    }
done
echo "VALUE_A=${VALUE_A}"
echo "VALUE_B=${VALUE_B}"

rm -r "${tmpdir}"

This example uses wait_file helper, but you might use inotifywait if you don't mind some dependencies on OS.