I have a Lambda function that needs to read a file from S3 every time it is executed.
The file is very small, about 200 bytes, the S3 bucket is in the US Standard region, the Lambda function is in the us-east-1 region (so the same region). It takes between 10 and 15 seconds to read the file, how come this is so slow?
Thanks.
EDIT: some code
long start = System.nanoTime();
AmazonS3Client s3Client = new AmazonS3Client();
S3Object propertyFile = null;
try {
propertyFile = s3Client.getObject(S3_BUCKET_NAME, S3_PROPERTY_FILE);
} catch (Exception e) {...}
try (InputStream in = propertyFile.getObjectContent()) {
PROPERTIES.load(in);
} catch (Exception e) {...}
LOGGER.debug("S3 access " + (System.nanoTime() - start));
EDIT #2: After Brooks' suggestion I did
AmazonS3Client s3Client = new AmazonS3Client(new InstanceProfileCredentialsProvider());
And I get this error:
Unable to load credentials from Amazon EC2 metadata service
EDIT #3:
The memory allocated to the Lambda function was 256MB, when I allocate 1024MB, it takes 3-4 seconds which is still too slow (it takes about 1-2 seconds when I test locally from my computer).
Lambda: Execution takes too long If your code takes much longer to run in Lambda than on your local machine, it may be constrained by the memory or processing power available to the function. Configure the function with additional memory to increase both memory and CPU.
You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes.
Create a Lambda Function to transform data for your use case. Create an S3 Object Lambda Access Point from the S3 Management Console. Select the Lambda function that you created above. Provide a supporting S3 Access Point to give S3 Object Lambda access to the original object.
Each S3 operation is an API request with significant latency — tens to hundreds of milliseconds, which adds up to pretty much forever if you have millions of objects and try to work with them one at a time.
What I would suggest is to use ProfileCredentialsProvider
and to cache S3 client instance between Lambda function executions:
public class MyLambda { // No matter you implement standard AWS SDK interfaces or not
private final AmazonS3Client s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
public String sayHello(Request request, Context context) {
S3Object s3Obj = s3Client.getObject(request.getBucket(), request.getKey());
return S3Utils.getContent(s3Object); // Some util which retrieves object content
}
}
The thing is that it takes some time to set up the S3 client which manages connections pool and other resources.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With