Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pandas in AWS lambda gives numpy error

I've been trying to run my code in AWS Lambda which imports pandas. So here is what I've done. I have a python file which contains a simple code as follows(This file has the lambda handler)

import json print('Loading function') import pandas as pd def lambda_handler(event, context):     return "Welcome to Pandas usage in AWS Lambda" 
  1. I have zipped this python file along with numpy, pandas and pytz libraries as a deployment package (Did all these in Amazon EC2 linux machine)
  2. Then uploaded the package into S3
  3. Created a lambda function(runtime=python3.6) and uploaded the deployment package from S3

But when I test the lambda function in AWS Lambda, I get the below error:

Unable to import module 'lambda_function': Missing required dependencies ['numpy'] 

I already have numpy in the zipped package but still I get this error. I tried to follow the hints given at Pandas & AWS Lambda but no luck.

Did anyone ran into the same issue. Would appreciate any hint or suggestions to solve this problem.

Thanks

like image 544
Kingz Avatar asked May 09 '17 18:05

Kingz


People also ask

Is NumPy available in AWS Lambda?

AWS Lambda does not include Pandas/NumPy Python libraries by default.

Can we use pandas in AWS Lambda?

pandas library is by default not available in AWS Lambda Python environments. If you try to import pandas in aws lambda function, you will get below error. For using pandas library in Lambda function a Lambda Layer needs to attached to the Lambda function.

Does NumPy work with pandas?

Introducing NumPy and PandasPandas is a high-level data manipulation tool that is built on the NumPy package. The key data structure in Pandas is called the DataFrame. DataFrames are incredibly powerful as they allow you to store and manipulate tabular data in rows of observations and columns.


1 Answers

EDIT: I figured out finally how to run pandas & numpy in a AWS Lambda python 3.6 runtime environment.

I have uploaded my deployment package to the following repo:

git clone https://github.com/pbegle/aws-lambda-py3.6-pandas-numpy.git

Simply add your lambda_function.py to the zip file by running:

zip -ur lambda.zip lambda_function.py

Upload to S3 and source to lambda.

ORIGINAL:

The only way I have gotten Pandas to work in a lambda function is by compiling the pandas (and numpy) libraries in an AWS Linux EC2 instance following the steps from this blog post and then using the python 2.7 runtime for my lambda function.

like image 81
0xPeter.eth Avatar answered Sep 27 '22 19:09

0xPeter.eth