Pandas & AWS Lambda

Question

Does anyone have a fully compiled version of pandas that is compatible with AWS Lambda?

After searching around for a few hours, I cannot seem to find what I'm looking for and the documentation on this subject is non-existent.

I need access to the package in a lambda function however I have been unsuccessful at getting the package to compile properly for usage in a Lambda function.

In lieu of the compilation can anyone provide reproducible steps to create the binaries?

Unfortunately I have not been able to successfully reproduce any of the guides on the subjects as they mostly combine pandas with scipy which I don't need and adds an extra layer of burden.

Answer 1

I believe you should be able to use the recent pandas version (or likely, the one on your machine). You can create a lambda package with pandas by yourself like this,

1. First find where the pandas package is installed on your machine i.e. Open a python terminal and type

   import pandas pandas.__file__ 

   That should print something like '/usr/local/lib/python3.4/site-packages/pandas/__init__.py'

2. Now copy the pandas folder from that location (in this case '/usr/local/lib/python3.4/site-packages/pandas) and place it in your repository.

3. Package your Lambda code with pandas like this:

   zip -r9 my_lambda.zip pandas/ zip -9 my_lambda.zip my_lambda_function.py 

You can also deploy your code to S3 and make your Lambda use the code from S3.

aws s3 cp  my_lambda.zip s3://dev-code//projectx/lambda_packages/ 

Here's the repo that will get you started

Answer 2

After some tinkering around and lot's of googling I was able to make everything work and setup a repo that can just be cloned in the future.

Key takeaways:

1. All static packages have to be compiled on an ec2 amazon Linux instance
2. The python code needs to load the libraries in the lib/ folder before executing.

Github repo: https://github.com/moesy/AWS-Lambda-ML-Microservice-Skeleton