How do you increase the token expiration time for a boto3.s3.transfer download?

Question

I am using boto3's s3.transfer to download several 4GB+ files from s3. All but one were able to download, but the one that failed gave the following error:

ERROR: An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.

I am using it the same way it is document at http://boto3.readthedocs.org/en/latest/_modules/boto3/s3/transfer.html

s3_client = session.client('s3')
transfer = S3Transfer(s3_client)
# Download s3://bucket/key to /tmp/myfile
transfer.download_file('bucket', 'key', '/tmp/myfile')

Is there a way to increase the expiration time of the signed url used inside boto3?

In case it is relevant, I am using Cognito to get the credentials, and with them, a session

    client = boto3.client('cognito-identity', AWS_REGION)

    # credentials[] contains the IdentityId and Token I get from my server
    # which I get using client.get_open_id_token_for_developer_identity 
    # with TokenDuration=86400
    resp = client.get_credentials_for_identity(IdentityId=credentials['IdentityId'],
                                               Logins={'cognito-identity.amazonaws.com': credentials['Token']})

    # The resp contains the actual temporary AWS secret/access codes and a session token, to be
    # used with the rest of the AWS APIs
    secretKey = resp['Credentials']['SecretKey']
    accessKey = resp['Credentials']['AccessKeyId']
    sessionToken = resp['Credentials']['SessionToken']

    session = Session(aws_access_key_id=accessKey,
                      aws_secret_access_key=secretKey,
                      aws_session_token=sessionToken,
                      region_name=AWS_REGION)

    s3_client = session.client('s3')

Answer 1

The problem you are experiencing is not linked to S3 signed URL as you are supposing.

Cognito is build on top of an IAM service called Security Token Service (STS). This service allows to generate temporary credentials (access key and secret key) by assuming a role (IAM user, EC2 instance, Lambda function etc ...) or by providing Web Identity token, for federated identities scenarios, using Google, Facebook, Amazon.

These credentials are limited in scope (to whatever IAM Role you have defined) and in time, between 15 secs and several hours, depending on the use case.

The credentials you are obtaining through Cognito are generated by STS. At low level, STS API allows to specify how long you want these credentials to stay valid (see http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html) . However, I can not find an equivalent in Cognito API (https://docs.aws.amazon.com/cognitoidentity/latest/APIReference/API_GetCredentialsForIdentity.html). I would love to be proven wrong on that point.

Why do you see the error ?

My guess, based on the elements you provided, is that your download code is running longer than the life of the temporary credentials you received. First download works, but later does not.

How to workaround it ?

• At low level, a clean solution would be to use STS instead of Cognito, but that would require a huge amount of work and your will loose all the benefits using Cognito (stable user ID across login providers, multiple login providers, unauthenticated users ...)
• Another solution, assuming you have multiple file transfers, in a loop, would be to check credentials expiration time, and renew them in between file transfer. Check resp['Credentials']['Expiration'] for the expiration time. You can renew Cognito provided credentials by calling get_credentials_for_identity again.
• You can also consider downloading your file through one of the 50+ edge locations provided by AWS. This is a new S3 capability, just released this week, that considerably speed up the upload or download of large files. See http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html for more details. There is price tag associated to that usage, see http://aws.amazon.com/s3/pricing/