Menu
I have a lambda function that reads from DynamoDB and creates a large file (~500M) in /tmp that finally uploaded to s3. Once uploaded the lambda clears the file from /tmp (since there is a high probability that the instance may be reused)
This function takes about 1 minute to execute, even if you ignore the latencies.
In this scenario, when i try to invoke the function again, in < 1m, i have no control if i will have enough space to write to /tmp. My function fails.
Questions: 1. What are the known work arounds in these kind of scenario? (Potentially give more space in /tmp or ensure a clean /tmp is given for each new execution) 2. What are the best practices regarding file creation and management in Lambda? 3. Can i attach another EBS or other storage to Lambda for execution ? 4. Is there a way to have file system like access to s3 so that my function instead of using /tmp can write directly to s3?
904
Service-linked role created for LambdaThe following service-linked role is automatically created in your AWS account when registering Lambda resources as scalable targets with Application Auto Scaling. This role allows Application Auto Scaling to perform supported operations within your account.
Lambda is engineered to provide managed scaling in a way that does not rely upon threading or any custom engineering in your code. As traffic increases, Lambda increases the number of concurrent executions of your functions.
It "scales" automatically by running in parallel on AWS infrastructure. You only pay while a function is running, per 100ms. It is the job of AWS to ensure that their back-end infrastructure scales to support the number of Lambda functions being run by all customers in aggregate.
Reserved concurrency – To make sure that your function can always scale to handle additional requests, you can reserve concurrency for it. Setting reserved concurrency for a function ensures that it can scale to, but not exceed, a specified number of concurrent invocations.
I doubt that two concurrently running instances of AWS Lambda will share /tmp or any other local resource, since they must execute in complete isolation. Your error should have a different explanation. If you mean, that a subsequent invocation of AWS Lambda reuses the same instance, then you should simply clear /tmp on your own.
In general, if your Lambda is a resource hog, you better do that work in an ECS container worker and use the Lambda for launching ECS tasks, as described here.
67
You are likely running into the 512 MB /tmp limit of AWS Lambda.
You can improve your performance and address your problem by storing the file in-memory, since the memory limit for Lambda functions can go as high as 1.5 GB.
45
Starting March 2022, Lambda now supports increasing /tmp directory's maximum size limit up to 10,240MB.
More information available here.
37
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
