AWS - boto.connect_s3() reporting "No handler was ready to authenticate" while EC2 instance is booting

Question

I'm trying to create a bootstrap script to setup EC2 instances. I want the instances to automatically download source code from an S3 bucket during boot. I recently read about assigning an IAM role to my EC2 instances so they can get temporary credentials from the metadata server. This way, I don't have to store AWS credentials on my image or pass them around during boot.

I wrote a Python script that performs the download by connecting to S3 using boto. I hooked up that script to the init.d config. My script works fine when I run it manually, in a remote SSH session, but for some reason it fails when it's executed while booting.

My code uses boto in the following manner:

import boto
try:
    conn = boto.connect_s3()
except:
    # log error
    exit(-1)

In theory, with boto 2.6, that code should contact the AWS metadata server to get temporary security credentials based on the IAM Role of the current EC2 instance.

For some reason, while booting, boto.connect_s3() raises a boto.exception.NoAuthHandlerFound: "No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials".

I traced the error down to the boto.auth module, in the get_auth_handler function (the only place this exception is raised). When the HmacAuthV1Handler object is created, the HmacKeys base class constructor fails if either the access key id or the secret key is None. Indeed, after a bit of experimentation in my script, I found that if I explicitly create an instance of the boto.provider.Provider class with name='aws' and no credentials, the Provider instance's access_key/secret_key fields both remain None (but only during boot; the credentials are initialized if I run the script manually afterwards). Yet I know for a fact that the metadata server isn't the culprit (a GET indeed returns the expected credentials, even during boot).

So... What could be wrong? Could it be an issue in boto? A config issue? Or am I just doing something wrong? Heck, could it be that Ubuntu is not in a "proper state" for that kind of operations while it's booting?

--- EDIT ---

Thanks to garnaat's answer below, I found that the problem is that the python path (sys.path) is completely different during boot than during my remote SSH session.

While booting:

['/home/bitnami', 
'/usr/lib/python2.7', 
'/usr/lib/python2.7/plat-linux2', 
'/usr/lib/python2.7/lib-tk', 
'/usr/lib/python2.7/lib-old', 
'/usr/lib/python2.7/lib-dynload', 
'/usr/local/lib/python2.7/dist-packages', 
'/usr/lib/python2.7/dist-packages']

and when running it from my SSH session:

['/home/bitnami', 
'/opt/bitnami/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', 
'/opt/bitnami/python/lib/python2.7/site-packages/MySQL_python-1.2.3c1-py2.7-linux-i686.egg', 
'/opt/bitnami/python/lib/python2.7/site-packages/html5lib-0.90-py2.7.egg', 
'/opt/bitnami/python/lib/python2.7/site-packages/boto-2.6.0-py2.7.egg', 
'/opt/bitnami/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', 
'/opt/bitnami/apps/django/lib/python2.7/site-packages', 
'/home/bitnami/deploy', 
'/opt/bitnami/python/lib/python27.zip', 
'/opt/bitnami/python/lib/python2.7', 
'/opt/bitnami/python/lib/python2.7/plat-linux2', 
'/opt/bitnami/python/lib/python2.7/lib-tk', 
'/opt/bitnami/python/lib/python2.7/lib-old', 
'/opt/bitnami/python/lib/python2.7/lib-dynload', 
'/opt/bitnami/python/lib/python2.7/site-packages', 
'/opt/bitnami/python/lib/python2.7/site-packages/PIL']

(I used the 32-bit BitNami DjangoStack 1.4.2-1 image as my base image.)

Answer 1

You mention that you are running on Ubuntu. I know that many Ubuntu distributions have an older version of boto pre-installed. I think it's possible that during the boot process it is finding and using that older version of boto which does not understand how to fetch the credentials from IAM roles.

To test this, try logging boto.Version in your script and see if it is 2.6.0 or not.