Reducing over 30 seconds cold start on AWS API Gateway + Lambda

Question

I've been facing an extremely slow cold start on Lambda Functions deployed in Docker containers together with an API Gateway.

Tech Stack:

• FastAPI
• Mangum (https://mangum.io/)
• API Gateway
• AWS Lambda

To do the deployment, I've been using AWS SAM with the following template file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  demo

Resources:
  AppFunction:
    Type: AWS::Serverless::Function
    Properties:
      Timeout: 118
      MemorySize: 3008
      CodeUri: app/
      PackageType: Image
      Events:
        ApiEvent:
          Properties:
              RestApiId:
                  Ref: FastapiExampleGateway
              Path: /{proxy+}
              Method: ANY
              Auth:
                ApiKeyRequired: true
          Type: Api
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: .
      
  FastapiExampleGateway:
    Type: AWS::Serverless::Api
    Properties:
        StageName: prod
        OpenApiVersion: '3.0.0'
        # Timeout: 30
    Auth:
      ApiKeyRequired: true
      UsagePlan:
        CreateUsagePlan: PER_API
        UsagePlanName: GatewayAuthorization

Outputs:
  Api:
    Description: "API Gateway endpoint URL for Prod stage for App function"
    Value: !Sub "https://${FastapiExampleGateway}.execute-api.${AWS::Region}.amazonaws.com/Prod/"

The lambda is relatively light, with the following requirements installed:

jsonschema==4.16.0
numpy==1.23.3
pandas==1.5.0
pandas-gbq==0.17.8
fastapi==0.87.0
uvicorn==0.19.0
PyYAML==6.0
SQLAlchemy==1.4.41
pymongo==4.3.2
google-api-core==2.10.1
google-auth==2.11.0
google-auth-oauthlib==0.5.3
google-cloud-bigquery==3.3.2
google-cloud-bigquery-storage==2.16.0
google-cloud-core==2.3.2
google-crc32c==1.5.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
mangum==0.11.0

And the Dockerfile I'm using for the deployment is:

FROM public.ecr.aws/lambda/python:3.9

WORKDIR /code

RUN pip install pip --upgrade

COPY ./api/requirements.txt /code/api/requirements.txt

RUN pip install --no-cache-dir -r /code/api/requirements.txt

COPY ./api /code/api

EXPOSE 7777

CMD ["api.main.handler"]

ENV PYTHONPATH "${PYTHONPATH}:/code/"

Leading to a 250mb image.

On the first Lambda pull, I'm seeing

which looks like it's a very long start before the actual lambda execution. It reaches the point where API gateway times out due to the maximum 30 second response!

• Local tests using sam local start-api work fine.
• I've tried increasing the lambda function RAM to higher values.

Not sure if this a problem with Mangum (wrapper for FastAPI)?

Answer 1

My experience with container-backed Lambdas is the same as yours. With a 1GB container my init time can be 5-20 seconds. Each library import takes about 10 times as long as it does on my personal machine, even with max memory(and therefore cpu) assigned.

There's a blog post from an AWS Lambda engineer here https://brooker.co.za/blog/2023/05/23/snapshot-loading.html that explains that AWS lazy loads the entire container file system from S3. What this means is that actual cold start times are much worse than they appear in synthetic tests (such as the blog post at mikhail.io/serverless/coldstarts/aws) because they may "use" a 5GB container, but I strongly suspect they don't touch any of the files in it.

This multi-stage cache that Lambda uses for the images also makes it hard to test any of this. Because although the logs tell you if you've had a cold start ("init" time), it doesn't tell you which Lambda cache it's pulling from (probably a mix of all three). When a function hasn't been used in 6 hours, I might get a cold start time of 2-5 seconds. But when it hasn't been used in a week, I'll get a cold start time of 10-20 seconds.

It's all pretty disappointing. The upshot is that you essentially must use provisioned concurrency for anything client-facing.