AWS Lambda, Python, Numpy and others as Layers

Question

I have been going at this for a while trying to get python, numpy and pytz added to AWS Lambda as Layers rather than having to zip and throw it at AWS with my .py file. I was able to follow multiple tutorials and all of them failed.

I have resorted to following this guide if I am to go with pandas, numpy or pytz for any functionality (AWS Lambda with Pandas and NumPy - Ruslan Korniichuk - Medium). So this is good but I do not want to have to recreate a zip each time if things change with my function needs ect. especially as my company is growing. We are simply trying to automate some tasks with Lamba using Cloudwatch to run jobs periodically. Nothing spectacular and I know there may be route with S3 and other instances. However, I have been able to successfully create layers for other libraries except for Pandas, Numpy and Pytz.

So, I am worried about scalability with this method. I am working on a mac and I am not sure what else to do: I have tried using Docker, I have tried building from wheels. Is there any viable tutorials that explain how to do this in detail?

Here are some of the tutorials I have tried. Yes, it does not mean I followed them correctly but I did not succeed in the end with most of them:

• AWS Lambda with Pandas and NumPy - Ruslan Korniichuk - Medium
• Introduction to Amazon Lambda, Layers and boto3 using Python3
• Creating New AWS Lambda Layer For Python Pandas Library

You name it and I might have already went through the steps, especially in these articles, to complete this task. And a lot of Stack question comments as well, which have been very helpful and insightful.

Thanks in advance for any advice, just here to learn!

Answer 1

This is probably not the answer you want to hear, but honestly the pain around getting certain compiled libraries into lambda layers was enough for my company to just stop using them. Instead we use tend to use either fargate or ECS with docker containers.

Besides the issues of compiling packages for lambdas, we also ran into major issues with the max size of lambdas. We regularly were hitting that cap and having to get more and more hacky to remove files in order to make them fit.

Update: AWS now lets you run Lambdas from containers in ECR, which solves this problem nicely.

Answer 2

You should not need to recompile the layer every time you deploy. We have a lambda layer specifically for ML libraries like numpy, pandas, and fbprophet. It works great b/c our lambda deployment zip files are tiny, speeding up development and deployment.

I'm happy to help further. Can you give more information about what you tried and what was going wrong?