How to parallelize a bash script containing nested for loops on huge dataset with GNU parallel?

Question

I'm actually launching many connections to IP addresses thanks to OpenSSL in order to detect if a server accept or not a specified cipher. I'm launching this script on 1 000 000 servers (which are contained in "listeIpShuffle.txt"). So, my script contains 2 for loop: the first one is used to get each line of my file containing IP addresses, and the second one to test for each cipher available in my OpenSSL version if the server accept or reject it.

I saw on GNU parallel's doc that it is possible to parallelize these kind of loop:

  *(for x in `cat xlist` ; do
    for y in `cat ylist` ; do
      do_something $x $y
    done
  done) | process_output*

... can be written like this:

*parallel do_something {1} {2} :::: xlist ylist | process_output*

But I'm failing trying to apply this on my script... Indeed my file containing my IP addresses is too big, and I get the famous "Too many arguments" error... How can I handle this problem and make my script parallel?

Thanks in advance!

Here is my script :

#!/usr/bin/env bash

#on recupere toutes les ciphers d'openssl
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')
fichier="./serveursMail/listeIpShuffle.txt"
port=":25"
>resultDeprecatedCipher.txt
nbInconnu=0
echo Cipher list de $(openssl version).

for ligne in $(<$fichier)
do
    ligneIp=$(echo $ligne | tr "|" "\n")    
    ip=($ligneIp)
    ipPort=$ip$port
    dns=$(echo $ligneIp |cut -f 2 -d ' ')


    ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')

    for cipher in ${ciphers[@]}
    do
        if [[ $nbInconnu < 4 ]] ; then
            echo -n Test $ligneIp " : " $cipher...

            result=$(echo -n | timeout 10s openssl s_client -starttls smtp -cipher "$cipher" -connect $ipPort -servername $dns 2>&1) #pas de reponse apres dasn les 15sec => FAIL

            if [[ "$result" =~ ":error:" ]] ; then
            error=$(echo -n $result | cut -d':' -f6)
            echo NON \($error\)
            let "nbInconnu=0"
            else
                if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher    :" ]] ; then
                    echo OUI
                    let "nbInconnu=0"
                    echo $ligneIp " : " $cipher >> resultDeprecatedCipher.txt
                else
                    echo REPONSE INCONNUE
                    let "nbInconnu++" #incrementation
                    echo $nbInconnu
                    echo $result
                fi
            fi
        else
          let "nbInconnu=0"
          break
        fi
    done 
done

Answer 1

echo alt3.gmail-smtp-in.l.google.com > serverlist
echo fo-ds-ats.member.g02.yahoodns.net >> serverlist

doit() {
    ip="$1"
    port="$2"
    cipher="$3"
    openssl s_client -starttls smtp -cipher "$cipher" -connect $ip:$port -servername $ip < /dev/null
}
export -f doit
parallel --tag --timeout 10 --retries 4 doit $1 :::: serverlist ::: 25 ::: $(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g') >tmp_results
# Post process tmp_results as needed

To not overload a single server add --shuf. To run as many as possible in parallel add -j0.