BASH...GNU Parallel

Question

Racking my brains and scraping Google to find a way to get the sem command with timeout option to give me an exit code on timeout. It doesn't have to be through sem, it's just a pain because I need to end up applying actions differently based on whether:

• I get an OK response from the script being called

• I get anything but OK from the script being called

• The call times out

The semaphore is used in this case to throttle calls to a script which I pass a list of filenames to for an outgoing message queue. As you can see in the rough test example below I'm able to able to deal with the first two scenarios, but I can't for the life of me get anything outputted that I can use from the process being terminated by timeout (i.e. outputtest.sh has sleep 6; echo testings)

sem --jobs 3 --timeout 3 -u \
'runproc=$(bash outputtest.sh q_xxxx); if [ "$runproc" == "00" ]; 
then echo "OK"; else mv ./q_xxxx ./err/err_xxxx; fi'

Any hints?

Answer 1

GNU Parallel waits until a semaphore is available, then it forks into the background and runs the job - possibly killing it if it times out.

So sem returns before the job is done and before it is timed out. Therefore sem cannot tell you whether the job will fail or time out at some later point in time.

A possible workaround is having a canary so you can see if the job completes or not:

rm canary
sem 'your_job; echo the job completed > canary'

If the sem is started from different parallel running scripts, then --fg may be your solution: It will block until the job has finished and give an exit value of 1 if the job fails/timesout, but other parallel running scripts will be able to use the same semaphore.

An alternative would be --joblog from which you can read the exit value (A timeout would give -1 and 15 as signal). But that does not work well with sem, so you would have to rewrite your program to use parallel (without --semaphore) instead.

To process a message queue you might want to look this for inspiration: http://www.gnu.org/software/parallel/man.html#example__gnu_parallel_as_queue_system_batch_manager