AWS Lambda Sporadic "No space left on device" error

Question

Amazon Lambda function invocations sporadically fail with no space left on device when downloading 20M file to /tmp. The lambda limits page clearly says that the limit for Ephemeral disk capacity (“/tmp” space) is 512M, presumably per lambda invocation, not per lambda function across all its invocation.

Below are the details:

A lambda function (java 8 runtime, role "lambda_basic_execution") is called concurrently (40 concurrent invocations). Each invocation downloads data from s3 (in all cases the data is LESS than the 512M limit). Randomly, 4 to 5 of these invocation fail with no left space on device error. I believe some of these invocation end up in the same machine and the same JVM, sharing their limit of 512M ephemeral disk. Sounds like a bug to me.

Here is the stack trace:

.....Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) at com.amazonaws.services.s3.internal.ServiceUtils.downloadToFile(ServiceUtils.java:295) ... 5 more

Any advice or workarounds will be greatly appreciated.

Cross posted here: https://forums.aws.amazon.com/thread.jspa?threadID=209428

Answer 1

The AWS Lambda FAQ says:

Each Lambda function receives 500MB of non-persistent disk space in its own /tmp directory.

It is possible that your environment is being reused, so your temp files might be accumulating and eventually passing the limit:

To improve performance, AWS Lambda may choose to retain an instance of your function and reuse it to serve a subsequent request, rather than creating a new copy. Your code should not assume that this will always happen.

Therefore, try removing temp files when they are no longer required. You can also insert some debugging code to list what might be inside the /tmp directory to identify the cause.