Menu
I have 4 lambda functions which will be invoked at same time (by SNS), the frequence of SNS's event is 5 minutes. Each function process the large mount of data and images(~300MB) so I store them on /tmp folder (500MB limit).
At the beginning of function, I wrote some code to the clean up /tmp folder, to make sure it's not out of memory (Because I've known that AWS lambda sometimes reuses previous container to improve performance).
I check it manually (create message and publish by SNS to 4 lambda functions), it worked fine.
But when it runs automatically (invoked each 5 minutes) the result is not as my expectation. The first execution is fine, but the next times after, 1 of 4 or even 4 lambda functions throw out the error related to "out of memory": "No space left on device", cannot load lib, ...
Previous, I use nodejs(4.3) it worked fine both case.
But I have to change to python for some reason, the main flow and the mount of created data is the same. But it's failed when run automatically.
I think that the issue came from the cache of previous container (reused container), I checked the /tmp after clean (ls -alh /tmp) there's no files but when check the storage (df /tmp) it show that used is 77%.
Any suggestion to make clean /tmp folder or work around solution is very appreciate. Thank!
Edited: Code I use to clean /tmp folder:
from subprocess import call ... call('rm -rf /tmp/*', shell=True) 
860
Data sharing: each Lambda function worker has its own instance of /tmp directory and they don't share any data.
Temporary storage with /tmp This space has a fixed size of 512 MB. The same Lambda execution environment may be reused by multiple Lambda invocations to optimize performance. The /tmp area is preserved for the lifetime of the execution environment and provides a transient cache for data between invocations.
Since the Lambda Function is just running on a container in a linux environment, we can use the OS to tell us generally how much space we have left on the temporary filesystem. You can see that at the end of the string is there the /tmp partition is located.
Yes, lambda being a managed service; they do reuse the same underlying resource if the lambda is getting invoked repeatedly. This was a production problem we faced and fixed by deleting the /tmp. On a separate note AWS should mention this in their FAQs.
if os.path.exists(tmp_file_path): os.remove(tmp_file_path) print("Removed the file %s" % tmp_file_path) else: print("Sorry, file %s does not exist." % tmp_file_path) If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
