How to parallelise a while loop in bash?

Question

I have the following while loop that runs in parallel. ( The logProcess is a function that I define earlier in my script. )

while read LINE; do
    logProcess $LINE &
done <<< "$ELS_LOGS"
wait

I need to find a way to limit the number of processes running. I know there are parallel processes running. How do I convert the loop to use that command?

Answer 1

May I recommend GNU Parallel to you. Your entire script would become:

parallel -a "$ELS_LOGS" logProcess

If logProcess is a function defined within your script, you will need to export it first before running GNU Parallel, like this:

export -f logProcess

Then if you want, say, 8 running at a time, you simply do:

parallel -j 8 -a "$ELS_LOGS" logProcess

If you want to see what it would do, without actually doing anything:

parallel --dry-run ...

If you want a progress bar, or an ETA:

parallel --eta ...
parallel --bar ...