Clearing out tmp folder from AWS Lambda

Question

Hi I have an AWS Lambda environment where the temp directory is now full and I get the following:

java.lang.RuntimeException: java.nio.file.FileSystemException: /tmp/out3786803744412914689: No space left on device

It's serverless so I cannot simply log into the box and delete the contents of the directory.

Is there any way to fix this other than deploying a code change to clear out the temp folder on restart?

Answer 1

To Add to @John Rotenstein's answer, our lambdas download a large ML model and move to /tmp at the start of the invocation.

In python we do something along the lines of:

if not os.path.isdir(f'/tmp/{self.model}'):
    self.download_model()

For our use case this is better than clearing the /tmp dir at the end of the lambda run as it reduces the number of calls and downloads required to/from s3, giving a performance boost for warm starts. It also means the lambdas will finish quicker as they don't need to cleanup. The caveat here is our model is static so we don't need to worry about cache invalidation. If you need to load frequently changing data then of course clear the /tmp dir.

Answer 2

When an AWS Lambda function is triggered, a temporary container is created. The Lambda function is then run within the container.

If the Lambda function is triggered many times, it is possible that multiple containers could be created. For example, if the function takes 5 seconds to run and 10 functions are triggered in one second, then 50 containers might be provisioned.

Also, once a function has completed executing, the container might be kept around and used again if the Lambda function is triggered again.

So, there is no single 'server' that is used for the Lambda function. It might be many, or it might be one that is reused.

It is recommended that functions delete their temporary files from /tmp before ending execution. This way, the space will be available for the next execution.

Conversely, you might want to intentionally keep some data in the container for the next execution to act like a cache. For example, if the function downloads some reference data, it will not need to re-download the data the next time if the container is reused.

Bottom line: Program the function to clean-up after itself.