AWS secret manager time out sometimes

Question

I am fetching a secret from secret manager on a lambda. The request fails sometimes. Which is totally strange, it is working fine and couple of hours later I check and I am getting time out.

def get_credentials(self):
    """Retrieve credentials from the Secrets Manager service."""

    boto_config = BotoConfig(connect_timeout=3, retries={"max_attempts": 3})
    secrets_client = self.boto_session.client(
        service_name="secretsmanager",
        region_name=self.boto_session.region_name,
        config=boto_config,
    )
    secret_value = secrets_client.get_secret_value(SecretId=self._secret_name)

    secret = secret_value["SecretString"]

I try to debug the lambda and later seems to be working again, without any change, those state changes happen in hours. Any hint why that could happen?

Traceback (most recent call last):
  File "/opt/python/botocore/endpoint.py", line 249, in _do_get_response
    http_response = self._send(request)
  File "/opt/python/botocore/endpoint.py", line 321, in _send
    return self.http_session.send(request)
  File "/opt/python/botocore/httpsession.py", line 438, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://secretsmanager.eu-central-1.amazonaws.com/"

Answer 1

You are using the legacy retry mode (is the default one in boto3), which has very limited functionality as it only works for a very limited number of errors.

You should try changing your retry strategy to something like Standard retry mode or Adaptative. In that case your code would look like:

from botocore.config import Config as BotoConfig
boto_config = BotoConfig(
    connect_timeout=3,
    retries={
        "max_attempts": 3,
        "mode":"standard"
    }
)
secrets_client = self.boto_session.client(
    service_name="secretsmanager",
    region_name=self.boto_session.region_name,
    config=boto_config,
)