Fabric - ThreadingGroup exception stops remaining requests?

Question

I'm new to Fabric, and would like to execute a series of commands against some remote SSH servers in parallel.

It seems that I should use a ThreadingGroup to do this, which I can do, and seems to work.

My only real problem is that I'd like to understand how to handle error cases, and how to pass in the server list as a string, which I'd get from a file or the command line.

How can I do this?

Answer 1

I found an example in a github issue, and extended it to work for my use case... hope it helps!

test.py

# requires fabric 2.x - run 'pip install fabric' to install it
import logging, socket, paramiko.ssh_exception
from fabric import Connection, Config, SerialGroup, ThreadingGroup, exceptions, runners
from fabric.exceptions import GroupException

# Note: You need to supply your own valid servers here to ssh to of course!
def main():
    testHosts("All should succeed", "validServer1,validServer2,validServer3")
    testHosts("Some should fail", "validServer1,validServer2,BADSERVER1,validServer3,BADSERVER2")

def testHosts(message, hostsAsString):
    print("")
    print(message)

    # Get list of hosts from somewhere, and convert them to connections
    hosts = hostsAsString.split(",")
    servers = [Connection(host=host) for host in hosts]
        
    # Create a thread group to run requests in parallel
    g = ThreadingGroup.from_connections(servers)
    try:
        command = "df -h / | tail -n1 | awk '{print $5}'"
        results = g.run(command, hide=True)
        for r in results:
            connection = results[r]
            print("{}".format(r.host) )
            print("  SUCCESS, " + connection.stdout.strip())
    except GroupException as e:
        # If an exception occurred, at least one request failed. 
        # Iterate through results here
        for c, r in e.result.items():
            print("{}".format(c.host) )
            if isinstance(r,runners.Result) :
                print("  SUCCESS, " + r.stdout.strip())
            elif isinstance(r,socket.gaierror) :
                print("  FAILED,  Network error")
            elif isinstance(r,paramiko.ssh_exception.AuthenticationException) :
                print("  FAILED,  Auth failed")
            else:
                print("  FAILED,  Something other reason")

main()

which produces the following output

$ python test.py

All should succeed
validServer1
  SUCCESS, 59%
validServer2
  SUCCESS, 54%
validServer3
  SUCCESS, 53%

Some should fail
validServer1
  SUCCESS, 59%
validServer2
  SUCCESS, 54%
validServer3
  SUCCESS, 53%
BADSERVER1
  FAILED,  Network error
BADSERVER2
  FAILED,  Network error