How to make Fabric ignore offline hosts in the env.hosts list?

Question

This is related to my previous question, but a different one.

I have the following fabfile:

from fabric.api import *

host1 = '192.168.200.181'
offline_host2 = '192.168.200.199'
host3 = '192.168.200.183'

env.hosts = [host1, offline_host2, host3]
env.warn_only = True

def df_h():
    with settings(warn_only=True):
        run("df -h | grep sda3")

And the output is:

[192.168.200.199] run: df -h | grep sda3

Fatal error: Low level socket error connecting to host 192.168.200.199: No route to host

Aborting.

After the execution hits the offline server, it aborts immediately, regardless of the other servers in the env.hosts list.

I have used the env setting "warn_only=True", but maybe I'm using it improperly.

How can I modify this behavior so that it will only prints the error and continue executing?

Answer 1

As of version 1.4 Fabric has a --skip-bad-hosts option that can be set from the command line, or by setting the variable in your fab file.

env.skip_bad_hosts = True

Documentation for the option is here: http://docs.fabfile.org/en/latest/usage/fab.html#cmdoption--skip-bad-hosts

Don't forget to explicitly set the timeout value also.

Answer 2

According to the Fabric documentation on warn_only,

env.warn_only "specifies whether or not to warn, instead of abort, when run/sudo/local encounter error conditions.

This will not help in the case of a server being down, since the failure occurs during the SSH attempt before executing run/sudo/local.

One solution would be to create a function to check if each server is up prior to executing your tasks. Below is the code that I used.

from __future__ import print_function
from fabric.api import run, sudo, local, env
import paramiko
import socket

host1 = '192.168.200.181'
offline_host2 = '192.168.200.199'
host3 = '192.168.200.183'

env.hosts = [host1, offline_host2, host3]

def df_h():
    if _is_host_up(env.host, int(env.port)) is True:
        run("df -h | grep sda1")


def _is_host_up(host, port):
    # Set the timeout
    original_timeout = socket.getdefaulttimeout()
    new_timeout = 3
    socket.setdefaulttimeout(new_timeout)
    host_status = False
    try:
        transport = paramiko.Transport((host, port))
        host_status = True
    except:
        print('***Warning*** Host {host} on port {port} is down.'.format(
            host=host, port=port)
        )
    socket.setdefaulttimeout(original_timeout)
    return host_status