How to check if SSH connection was established with AWS instance

Question

I'm trying to connect to the Amazon EC2 instance via SSH using boto. I know that ssh connection can be established after some time after instance was created. So my questions are:

• Can I somehow check if SSH is up on the instance? (if so, how?)
• Or how can I check for the output of boto.manage.cmdshell.sshclient_from_instance()? I mean for example if the output prints out Could not establish SSH connection, than try again.

That's what I tried so far, but have no luck:

if instance.state == 'running':
    retry = True
    while retry:
        try:
            print 'Connecting to ssh'
            key_path = os.path.join(os.path.expanduser('~/.ssh'), 'secret_key.pem')
            cmd = boto.manage.cmdshell.sshclient_from_instance(instance,
                                                               key_path,
                                                               user_name='ec2-user')

            print instance.update()
            if cmd:
                retry = False
        except:
            print 'Going to sleep'
            time.sleep(10)

SSH Connection refused, will retry in 5 seconds
SSH Connection refused, will retry in 5 seconds
SSH Connection refused, will retry in 5 seconds
SSH Connection refused, will retry in 5 seconds
SSH Connection refused, will retry in 5 seconds
Could not establish SSH connection

And of course everything is working properly, because I can launch the same code after some time and will get a connection, and will be able to use cmd.shell()

Answer 1

The message "SSH Connection refused, will retry in 5 seconds" is coming from boto: http://code.google.com/p/boto/source/browse/trunk/boto/manage/cmdshell.py

Initially, 'running' just implicates that the instance has started booting. As long as sshd is not up, connections to port 22 are refused. Hence, what you observe is absolutely to be expected if sshd does not come up within the first 25 seconds of 'running' state.

Since it is not predictable when sshd comes up exactly and in case you do not want to waste time by just defining a constant long waiting period, you could implement your own polling code that in e.g. 1 to 5 second intervals checks if port 22 is reachable. Only if it is invoke boto.manage.cmdshell.sshclient_from_instance().

A simple way to test if a certain TCP port of a certain host is reachable is via the socket module:

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
    s.connect(('hostname', 22))
    print "Port 22 reachable"
except socket.error as e:
    print "Error on connect: %s" % e
s.close()

Answer 2

I have 2 parts, one to check if the instance is running, then another one to check if the instance is reachable

# Get instance status till it is running
status_output=$(aws ec2 describe-instance-status --instance-ids $instance_id)
instance_status=$(jq -n "$status_output" | jq .InstanceStatuses[0] | jq .InstanceState.Name)
echo $instance_status
while [ ${instance_status:1:-1} != running ]
do
    status_output=$(aws ec2 describe-instance-status --instance-ids $instance_id)
    instance_status=$(jq -n "$status_output" | jq .InstanceStatuses[0] | jq .InstanceState.Name)
    echo $instance_status
done

# Get instance reachability till it is ready 
status_output=$(aws ec2 describe-instance-status --instance-ids $instance_id)
instance_reachability=$(jq -n "$status_output" | jq .InstanceStatuses[0] | jq .InstanceStatus.Status)
echo $instance_reachability
while [ ${instance_reachability:1:-1} != ok ]
do
    status_output=$(aws ec2 describe-instance-status --instance-ids $instance_id)
    instance_reachability=$(jq -n "$status_output" | jq .InstanceStatuses[0] | jq .InstanceStatus.Status)
    echo $instance_reachability
done