Is there a way to use an AWS signed URL to perform a multipart upload?

Question

I'm writing a client for a service that provides a signed url for uploads. This works just fine for smaller uploads, but fails for larger uploads that would be benefit from using a multipart upload.

The authorization docs suggest that I can use the provided signature and access key id in both the URL or via the Authorization header. I"ve tried using the header approach to start the multipart upload, but I get an access denied. When I use the query string approach, I get a method not allowed (POST in this case).

I'm using boto to generate the URL. For example:

import boto

c = boto.connect_s3()
bucket = c.get_bucket('my-bucket')
key = boto.s3.key.Key(bucket, 'my-big-file.gz')
signed_url = key.generate_url(60 * 60, 'POST')  # expires in an hour

Then when trying to start the multipart upload using the signed URL, I'm doing the following:

import requests

url = signed_url + '&uploads'
resp = requests.post(url)

This returns a method not allowed.

Is this strategy possible? Is there a better way to provide limited credentials to a specific resource in order to allow large multipart uploads?

Update

I've managed to find a slightly more specific error that makes me think this isn't possible. Unfortunately, I get a 403 saying that the signature doesn't match the request.

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you
  provided. Check your key and signing method.</Message>
  <StringToSignBytes>.../StringToSignBytes>
  <RequestId>...</RequestId>
  <HostId>...</HostId>
  <SignatureProvided>...</SignatureProvided>
  <StringToSign>POST


  1402941975
  /my-sandbox/test-mp-upload.txt?uploads</StringToSign>
  <AWSAccessKeyId>...</AWSAccessKeyId>
</Error>

This makes me think that I won't be able to use the signed URL because the signature won't match.

UPDATE

I've decided that it is not reasonable to use a signed URL for a multipart upload. While I suspect it is technical possible, it is not practical. The reason being is that a signed URL requires the URL, headers and request method all match exactly in order to work as expected. As a multipart upload needs to initialize the upload, upload each part and finalize (or cancel) the upload, it would be some what painful to generate URLs for each step.

Instead, I found can create a federated token to provide read / write access to a specific key in a bucket. This ends up being more practical and simple because I can immediately use boto as though I had credentials.

Answer 1

Here is some code to save time for the next person who wants to do this:

import json
from uuid import uuid4

import boto3


def get_upload_credentials_for(bucket, key, username):
    arn = 'arn:aws:s3:::%s/%s' % (bucket, key)
    policy = {"Version": "2012-10-17",
              "Statement": [{
                  "Sid": "Stmt1",
                  "Effect": "Allow",
                  "Action": ["s3:PutObject"],
                  "Resource": [arn],
              }]}
    client = boto3.client('sts')
    response = client.get_federation_token(
        Name=username, Policy=json.dumps(policy))
    return response['Credentials']


def client_from_credentials(service, credentials):
    return boto3.client(
        service,
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
    )


def example():
    bucket = 'mybucket'
    filename = '/path/to/file'

    key = uuid4().hex
    print(key)

    prefix = 'tmp_upload_'
    username = prefix + key[:32 - len(prefix)]
    print(username)
    assert len(username) <= 32  # required by the AWS API

    credentials = get_upload_credentials_for(bucket, key, username)
    client = client_from_credentials('s3', credentials)
    client.upload_file(filename, bucket, key)
    client.upload_file(filename, bucket, key + 'bob')  # fails


example()

Answer 2

To answer my own question, the best option is to use the Security Token Service to generate a set of temporary credentials. A policy description can be provided to limit the credentials to the specific bucket and key.