Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Lambda function extremely slow to retrieve S3 file

I have a Lambda function that needs to read a file from S3 every time it is executed.
The file is very small, about 200 bytes, the S3 bucket is in the US Standard region, the Lambda function is in the us-east-1 region (so the same region). It takes between 10 and 15 seconds to read the file, how come this is so slow?

Thanks.

EDIT: some code

long start = System.nanoTime();
AmazonS3Client s3Client = new AmazonS3Client();
S3Object propertyFile = null;
try {
    propertyFile = s3Client.getObject(S3_BUCKET_NAME, S3_PROPERTY_FILE);
} catch (Exception e) {...}
try (InputStream in = propertyFile.getObjectContent()) {
    PROPERTIES.load(in);
} catch (Exception e) {...}
LOGGER.debug("S3 access " + (System.nanoTime() - start));

EDIT #2: After Brooks' suggestion I did

AmazonS3Client s3Client = new AmazonS3Client(new InstanceProfileCredentialsProvider());

And I get this error:

Unable to load credentials from Amazon EC2 metadata service

EDIT #3:
The memory allocated to the Lambda function was 256MB, when I allocate 1024MB, it takes 3-4 seconds which is still too slow (it takes about 1-2 seconds when I test locally from my computer).

like image 690
Maxime Laval Avatar asked Feb 09 '16 18:02

Maxime Laval


People also ask

Why is my Lambda function taking so long?

Lambda: Execution takes too long If your code takes much longer to run in Lambda than on your local machine, it may be constrained by the memory or processing power available to the function. Configure the function with additional memory to increase both memory and CPU.

How do I maximize the read speed on Amazon S3?

You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes.

How Lambda function read data from S3 bucket?

Create a Lambda Function to transform data for your use case. Create an S3 Object Lambda Access Point from the S3 Management Console. Select the Lambda function that you created above. Provide a supporting S3 Access Point to give S3 Object Lambda access to the original object.

Is S3 high latency?

Each S3 operation is an API request with significant latency — tens to hundreds of milliseconds, which adds up to pretty much forever if you have millions of objects and try to work with them one at a time.


1 Answers

What I would suggest is to use ProfileCredentialsProvider and to cache S3 client instance between Lambda function executions:

public class MyLambda { // No matter you implement standard AWS SDK interfaces or not

    private final AmazonS3Client s3Client = new AmazonS3Client(new ProfileCredentialsProvider());

    public String sayHello(Request request, Context context) {
        S3Object s3Obj = s3Client.getObject(request.getBucket(), request.getKey());
        return S3Utils.getContent(s3Object); // Some util which retrieves object content
    }
}

The thing is that it takes some time to set up the S3 client which manages connections pool and other resources.

like image 132
Nazar Avatar answered Sep 19 '22 13:09

Nazar