Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Bash threading: wait for all job threads to finish doesn't work?

I'm writing a little script, that will create archives in main thread and after each archive is complete, a new thread would be created by calling function that would take care of uploading these archives. The reason I want uploading to be done in background is so that another archive could be created while the previous archives are being uploaded.

The problem I'm having is at the very end of the script. That is, main thread don't wait for all uploading threads to finish before exiting. Look at the following simplified script (I removed/changed parts of the code not related to the problem)

function func {
for files in /home/somewhere/
  do
    echo "Uploading $1" &
  done
wait
}

find /home/some/path -type f | while read filename ; do
  echo "Creating archive of $filename"
  func $somevariable &
done

wait

Everything is executing very nicely until the last archive is created, then the script ends before all func threads finish, leaving many files not uploaded.

Thank you for your ideas.

like image 979
Gargauth Avatar asked Feb 14 '10 01:02

Gargauth


1 Answers

Update: good points in the comment.

So, on a second look, it turns out the problem is the subshell that is created by the pipe to the loop. It's a good way to structure the script but you need to do the final wait in the shell that spun off the background tasks.

So do something like this:

find /home/some/path -type f | (while read filename; do
    echo "Creating archive of $filename"
    func $somevariable &
  done
  wait
)
like image 112
DigitalRoss Avatar answered Sep 18 '22 13:09

DigitalRoss