Boto generate_url for S3 PUT doesn't work with IAM roles?

Question

I use the following Python/Boto code to generate one-time file-upload URLs to an Amazon S3 bucket:

from boto.s3.connection import S3Connection

def get_signed_upload_url():
    s3 = S3Connection(ACCESS_KEY_ID, SECRET_ACCESS_KEY, is_secure=True)
    return s3.generate_url(300, 'PUT', bucket=BUCKET, key=KEY,
        headers={'Content-Type': 'text/plain'})

It has worked fine for several years, and it continues to work today.

But now, I'm in the process of converting to IAM roles — which will save me from hard-coding the ACCESS_KEY_ID and SECRET_ACCESS_KEY. Hence, I have removed the hard-coded keys, resulting in this code:

from boto.s3.connection import S3Connection

def get_signed_upload_url():
    s3 = S3Connection(is_secure=True)
    return s3.generate_url(300, 'PUT', bucket=BUCKET, key=KEY,
        headers={'Content-Type': 'text/plain'})

With this code, whenever I generate a URL and do a PUT request to it from my client-side web app (via Ajax), I get an HTTP 400 Bad Request error from S3:

<Error>
    <Code>InvalidToken</Code>
    <Message>The provided token is malformed or otherwise invalid.</Message>
    <!-- account-specific stuff removed -->
</Error>

Why is this happening?

Some extra details:

• This is deployed to an EC2 server that gets its IAM role assigned automatically. I've confirmed the authorization from the role is correctly made available to my Python code. All other parts of my code — including generate_url() calls that generate GET URLs — work fine, using the ACCESS_KEY_ID passed automatically to the EC2 instance's environment.
• Python 2.7.9 and Boto 2.38.0.
• The IAM role appears to have all appropriate permissions, as evidenced by the fact that other parts of my code are successfully interacting with S3 (getting items and creating items) using the role. It's only this particular PUT that's not working.
• The front-end logic sends a Content-Type header, which is why that header is in there. I tried removing the Content-Type header from this Boto code and from the front-end logic, but the problem still happens.

Answer 1

For what it's worth, I'm not able to replicate this problem. I've set up a brand new EC2 instance, assigned it an IAM role which has the AmazonS3FullAccess permission attached

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

The earliest Python version I was able to install was 2.7.12 with Boto 2.38.0

[root@ip-172-31-13-14 ~]# python -V
Python 2.7.12
[root@ip-172-31-13-14 ~]# pip freeze|grep boto
boto==2.38.0
botocore==1.5.95

I then run the code app.py

from boto.s3.connection import S3Connection


def get_signed_upload_url():
    BUCKET = 'test'
    KEY = 'test-2'
    s3 = S3Connection(is_secure=True)
    return s3.generate_url(300, 'PUT', bucket=BUCKET, key=KEY,
        headers={'Content-Type': 'text/plain'})


if __name__ == "__main__":
    url = get_signed_upload_url()
    print "curl -v -X PUT -H 'Content-Type: text/plain' -d @hello2.txt '" + url + "'"

Then run the resultant cURL command and the file is uploaded successfully

> Host: test.s3.amazonaws.com
> User-Agent: curl/7.53.1
> Accept: */*
> Content-Type: text/plain
> Content-Length: 8585
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< x-amz-id-2: redacted
< x-amz-request-id: redacted
< Date: Tue, 19 Dec 2017 05:45:15 GMT
< x-amz-version-id: redacted
< ETag: "redacted"
< Content-Length: 0
< Server: AmazonS3

At this stage I can only guess that perhaps the IAM role you're using doesn't have sufficient privileges to perform the PUT action? Failing that, there could be a difference in the way your client side code is actually performing the PUT request, but this is not very likely.

Other things to check:

• Ensure the clock time on the EC2 instance is accurate
• Ensure the AWS_REGION environment variable is set