Loop through CURL results

Question

I have the following script:

#!/bin/bash
for line in $('curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | zcat | head -n 1000 | dap json + select vhost + lines');
do
    echo "$line\n"
done

For which I am trying to achieve the following:

1. Loop through a dynamically growing list of results from the scans.io curl
2. Output the line which I then propose to pass to PHP to store and process

However I'm getting syntax error near unexpected token$'\r''` but am not a BASH expert to know what I need to tweak.

Answer 1

Use Process-Substitution, with a while-loop, see why using for-loop for command output parsing is bad. The unsetting of IFS and read with the -r flag does not allow backslashes to escape any characters and treat strings "as-is".

#!/bin/bash

while IFS= read -r line
do
    printf '%s\n' "$line"
done < <(curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | zcat | head -n 1000 | dap json + select vhost + lines)