Issues trying to SSH into a fresh EC2 instance with Paramiko

Question

I'm working on a script that spins up a fresh EC2 instance with boto and uses the Paramiko SSH client to execute remote commands on the instance. For whatever reason, the Paramiko client is unabled to connect, I get the error:

Traceback (most recent call last):
  File "scripts/sconfigure.py", line 29, in <module>
    ssh.connect(instance.ip_address, username='ubuntu', key_filename=os.path.expanduser('~/.ssh/test'))
  File "build/bdist.macosx-10.3-fat/egg/paramiko/client.py", line 291, in connect
  File "<string>", line 1, in connect
socket.error: [Errno 61] Connection refused

I can ssh in fine manually using the same key file and user. Has anyone run into issues using Paramiko? My full code is below. Thanks.

import boto.ec2, time, paramiko, os
# Connect to the us-west-1 region
ec2 = boto.ec2.regions()[3].connect()
image_id = 'ami-ad7e2ee8'
image_name = 'Ubuntu 10.10 (Maverick Meerkat) 32-bit EBS'
new_reservation = ec2.run_instances(
    image_id=image_id,
    key_name='test',
    security_groups=['web'])

instance = new_reservation.instances[0]

print "Spinning up instance for '%s' - %s. Waiting for it to boot up." % (image_id, image_name)
while instance.state != 'running':
    print "."
    time.sleep(1)
    instance.update()

print "Instance is running, ip: %s" % instance.ip_address

print "Connecting to %s as user %s" % (instance.ip_address, 'ubuntu')
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(instance.ip_address, username='ubuntu', key_filename=os.path.expanduser('~/.ssh/test'))
stdin, stdout, stderr = ssh.exec_command('echo "TEST"')
print stdout.readlines()
ssh.close()

Answer 1

I seem to have figured this out by trial and error. Even though the instance status is "running" according to boto, there is a delay for when it will actually allow an SSH connection. Adding a "time.sleep(30)" before the "ssh.connect(...)" seems to do the trick for me, though this may vary.

Answer 2

The way to check it's ssh available is to make sure its two status checks both passes. On web UI it looks like this:

And using boto3 (the original question used boto but it was 5 years ago), we can do:

session = boto3.Session(...)
client = session.client('ec2')
res = client.run_instances(...) # launch instance
instance_id = res['Instances'][0]['InstanceId']

while True:
    statuses = client.describe_instance_status(InstanceIds=[instance_id])
    status = statuses['InstanceStatuses'][0]
    if status['InstanceStatus']['Status'] == 'ok' \
            and status['SystemStatus']['Status'] == 'ok':
        break
    print '.'
    time.sleep(5)
print "Instance is running, you are ready to ssh to it"

Answer 3

Why not use boto.manage.cmdshell instead?

cmd = boto.manage.cmdshell.sshclient_from_instance(instance,
                                                   key_path,
                                                   user_name='ec2_user')

(code taken from line 152 in ec2_launch_instance.py)

For available cmdshell commands have a look at the SSHClient class from cmdshell.py.

Answer 4

I recently ran into this issue. The "correct" way would be to initiate a close() first and then reopen the connection. However on older versions, close() was broken.

With this version or later, it should be fixed: https://github.com/boto/boto/pull/412

"Proper" method:

newinstance = image.run(min_count=instancenum, max_count=instancenum, key_name=keypair, security_groups=security_group, user_data=instancename, instance_type=instancetype, placement=zone)
time.sleep(2)
newinstance.instances[0].add_tag('Name',instancename)

print "Waiting for public_dns_name..."
counter = 0
while counter < 70:
    time.sleep(1)
    conn.close()
    conn = boto.ec2.connection.EC2Connection(ec2auth.access_key,ec2auth.private_key)
    startedinstance = conn.get_all_instances(instance_ids=str(newinstance.instances[0].id))[0]
    counter = counter + 1
    if str(startedinstance.instances[0].state) == "running":
        break
    if counter == 69:
        print "Timed out waiting for instance to start."
print "Added: " + startedinstance.instances[0].tags['Name'] + " " + startedinstance.instances[0].public_dns_name