How to cycle through a small set of options within a for loop?

Question

I have a bunch of jobs I need to submit to a job queue. The queue has 8 different machines I can pick from or I can submit to any available server. Sometimes a server may be faulty so I would like to be able to loop through available servers I send my jobs to. A barebones version is below

# jobscript.sh
dir='some/directory/of/files/to/process'
for fn in $(ls $dir); do
    submit_job -q server@machine -x python script.py $fn
done

If I don't care what machine the job is sent to I remove the @machine portion so the command is just submit_job -q server -x python script.py $fn.

If I do want to specify the specific machine then I specify which machine by appending a number after machine as server@machine1 then on the next iteration server@machine2 then server@machine2 etc. The output of the script would then look like the following if I only use the first 3 servers

submit_job -q server@machine1 -x python script.py file1
submit_job -q server@machine2 -x python script.py file2
submit_job -q server@machine3 -x python script.py file3
submit_job -q server@machine1 -x python script.py file4
submit_job -q server@machine2 -x python script.py file5
submit_job -q server@machine3 -x python script.py file6
submit_job -q server@machine1 -x python script.py file7
submit_job -q server@machine2 -x python script.py file8
...

The list of available servers is [1, 2, 3, 4, 5, 6, 7, 8] but I would like to additionally specify from the command line a list of servers to ignore so something like

$bash jobscript.sh -skip 1,4,8

which would only cycle through 2, 3, 5, 6, 7 and product the output

submit_job -q server@machine2 -x python script.py file1
submit_job -q server@machine3 -x python script.py file2
submit_job -q server@machine5 -x python script.py file3
submit_job -q server@machine6 -x python script.py file4
submit_job -q server@machine7 -x python script.py file5
submit_job -q server@machine2 -x python script.py file6
submit_job -q server@machine3 -x python script.py file7
submit_job -q server@machine5 -x python script.py file8
submit_job -q server@machine6 -x python script.py file8
...

if flag -skip is not present, just run the command without @machine which will allow the queue to decide where to place the job and the commands look like

submit_job -q server -x python script.py file1
submit_job -q server -x python script.py file2
submit_job -q server -x python script.py file3
submit_job -q server -x python script.py file4
submit_job -q server -x python script.py file5
submit_job -q server -x python script.py file6
submit_job -q server -x python script.py file7
submit_job -q server -x python script.py file8
submit_job -q server -x python script.py file8
...

Answer 1

Something like this should do most of the work for you:

#!/bin/bash

machines=(1 2 3 4 5 6 7 8)
skip_arr=(1 4 8)
declare -a arr
for i in "${machines[@]}"; do
    if [[ ! " ${skip_arr[@]} " =~ " $i " ]]; then
        arr+=($i)
    fi
done
arr_len="${#arr[@]}"
declare -i i=0
for f in $(ls); do
    i="i % arr_len"
    echo "file is $f, machine is $i"
    let i++
done

Right now, I've set it up to go through the current directory, and just echo the values of the machine and filename. Obviously you'll want to change this to actually execute the commands from the right directory.

The last thing you need to do is build up skip_arr from the command line input, and then check if it's empty when you're executing your command.

Hopefully this gets you most of the way there. Let me know if you have any questions about anything I've done here.

Answer 2

Cycle through array of machines

#!/bin/bash

rotate() {
   if [[ "$1" = "all" ]]; then
      machines=(1 2 3 4 5 6 7 8)
   else
      machines=($*)
   fi

   idx=0
   max=${#machines[@]}

   for ((fn=0; fn<20; fn++)); do
      if (( $max > 0 )); then
         servernr=${machines[idx]}
         ((idx=(idx+1) % ${max}))
      else
         servernr=""
      fi
      echo "submit -q server${servernr} file${fn}"
   done
}

# test
echo "Rotate 0 machines"
rotate
echo "Rotate all machines"
rotate all
echo "Rotate some machines"
rotate 2 5 6