Use bash wait in for-loop [duplicate]

Question

(I have searched and expected this question to have been asked before but couldn't find anything like this although there are plenty of similar questions)

I want this for-loop to run in 3 different threads/processes and wait seem to be the right command

for file in 1.txt 2.txt 3.text 4.txt 5.txt
        do something lengthy &
        i=$((i + 1))
        wait $!
done

But this construct, I guess, just starts one thread and then wait until it is done before it starts the next thread. I could place wait outside the loop but how do I then

1. Access the pids?
2. Limit it to 3 threads?

Answer 1

The jobs builtin can list the currently running background jobs, so you can use that to limit how many you create. To limit your jobs to three, try something like this:

for file in 1.txt 2.txt 3.txt 4.txt 5.txt; do
  if [ $(jobs -r | wc -l) -ge 3 ]; then
    wait $(jobs -r -p | head -1)
  fi

  # Start a slow background job here:
  (echo Begin processing $file; sleep 10; echo Done with $file)&
done
wait # wait for the last jobs to finish

Answer 2

The GNU Parallel might be worth a look.

My first attempt,

parallel -j 3 'bash -c "sleep {};   echo {};"' ::: 4 1 2 5 3

can be, according to the inventor of parallel, be shortened to

parallel -j3 sleep {}\; echo {} ::: 4 1 2 5 3
1
2
4
3
5

and masking the semicolon, more friendly to type, like this:

parallel -j3 sleep {}";" echo {} ::: 4 1 2 5 3

works too.

It doesn't look trivial and I only tested it 2 times so far, once to answer this question. parallel --help shows a source where there is more info, the man page is a little bit shocking. :)

parallel -j 3 "something lengthy {}" ::: {1..5}.txt

might work, depending on something lengthy being a program (fine) or just bashcode (afaik, you can't just call a bash function in parallel with parallel).

On xUbuntu-Linux 16.04, parallel wasn't installed but in the repo.

Answer 3

Building on Rob Davis' answer:

#!/bin/bash
qty=3

for file in 1.txt 2.txt 3.txt 4.txt 5.txt; do
    while [ `jobs -r | wc -l` -ge $qty ]; do
        sleep 1
        # jobs #(if you want an update every second on what is running)
    done
    echo -n "Begin processing $file"
    something_lengthy  $file &
    echo $!
done
wait