SQS ExpiredToken: The security token included in the request is expired status code: 403

Question

I have a long-running worker process running on EC2 that consumes items from an SQS queue. After a time (8-12 hours, I reckon) I begin getting expired security token errors. I would expect the aws lib to handle the refresh of credentials automatically but this seems not to be the case. Is it in anyway handled within the client? This happens only when I use the DefaultCredentialsProviderChain to generate the access. This error does not occur when used with key and secret. The stacktrace is as below:

com.amazonaws.AmazonServiceException: The security token included in the request is expired (Service: AmazonSQS; Status Code: 403; Error Code: ExpiredToken; Request ID: 6ff6e1a0-d668-5ac5-bcd7-ae30058f25c0)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2419)
    at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1130)
    at com.amazonaws.services.sqs.AmazonSQSAsyncClient$24.call(AmazonSQSAsyncClient.java:1783)
    at com.amazonaws.services.sqs.AmazonSQSAsyncClient$24.call(AmazonSQSAsyncClient.java:1779)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

The workaround I found was to renew the awsCredentials everytime I encountered the expired token error and reset the sqs client.

awsCredentials = (new DefaultAWSCredentialsProviderChain).getCredentials
sqs = SimpleSQSClient(awsCredentials, Regions.US_EAST_1)
queueSQS = sqs.simple(QueueName(queueName), true)

Note: I am using the wrapper kifi/franz

Answer 1

The AWS SDKs are indeed capable of cycling temporary credentials inherited from the instance profile, but by passing an explicit AWSCredentials object in the constructor of SimpleSQSClient I believe you are denying it the opportunity to do so.

You didn't explicitly state that your application is inheriting the instance role, but there is enough evidence in your post to infer that this is the case:

• Your application is running on EC2.
• DefaultAWSCredentialsProviderChain's behavior is to look for "Instance profile credentials delivered through the Amazon EC2 metadata service" if it can find no other credentials.
• You're only seeing this behavior when not explicitly passing your own known access/secret keys.

The specific behavior for automatic credentials refresh is described in the documentation:

The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials.

By passing AWSCredentials directly instead of an AWSCredentialsProvider, you become responsible for checking and refreshing expired credentials. On the plus side, your workaround is just fine if you want to keep passing credentials explicitly.

SimpleSQSClient has a constructor that will work better for your use case:

new SimpleSQSClient(
    credentialProvider: com.amazonaws.auth.AWSCredentialsProvider,
    region: com.amazonaws.regions.Regions,
    buffered: Boolean
)

Example:

SimpleSQSClient sqs = SimpleSQSClient(new DefaultAWSCredentialsProviderChain(), Regions.US_EAST_1, false)

Example, explicitly using InstanceProfileCredentialsProvider:

SimpleSQSClient sqs = SimpleSQSClient(new InstanceProfileCredentialsProvider(), Regions.US_EAST_1, false)

Further reading:

• AWS SDK for Java > DefaultAWSCredentialsProviderChain - Describes default provider chain in more detail
• kini/franz - Initialization - SimpleSQSClient Constructor References
• AWS SDK for Java > Developer Guide > Using IAM Roles to Grant Access to AWS Resources on Amazon EC2 - A great resource for correct usage of temporary credentials
• Update 2017/11/06: DefaultAWSCredentialsProviderChain was renamed to DefaultCredentialsProvider during the AWS SDK for Java 2.0 preview. Preview documentation here.