Add wait between parallel processes in bash

Question

I have a bash script to upload data to a site. I was getting slow upload speeds, so I started running it in parallel, 5 at the same time, using xargs and -N1.

However, the problem is that the server asks me to solve a captcha if I run it 5 at a time, whereas it works fine with 1 at a time.

I figure this is because all the processes start at exactly the same time, I'm getting flagged.

Anyway so here's the question, is there any way for me to add a wait (say 1 second) between starting processes in xargs/gnu parallel?

The only thing I could come up with is using pgrep script | wc -1 to count the script instances, and sleep for that number of seconds.

However, this is really not optimal, are there any better ways of doing this?

Answer 1

If the upload takes a random amount of time you just need the first 5 to start with a 1-5 second delay:

cat list | parallel -j5 [ {#} -lt 6 ] \&\& sleep {#}\; upload {}

Answer 2

Rather than using xargs, I think you just want a loop, as in

for i in {1..5}; do sleep 5; your-command & done

This forks off the commands every 5 seconds. For an increasing delay (if that's needed):

for i in {1..5}; do ((w=i*5)); sleep $w; your-command & done

Another alternative:

files="a.txt b.txt c.txt"
for i in $files; do upload-command $i& sleep 5; done