AWS Lambda Python3.7 Function - numpy: cannot import name 'WinDLL'

Question

I have a function set up in lambda that runs a python script from a .zip file. I have created a virtualenv and included all of the necessary packages in the .zip file (from the Lib\site-packages folder).

Below are the import statements for the packages used in the script:

import requests
import boto3
import logging
import os
from botocore.exceptions import ClientError
from pprint import pprint
import pandas as pd
from datetime import datetime
import s3fs

When I attempt to run the lambda function I am receiving the following error:

START RequestId: e302cee0-3c51-453a-84c1-6eb1f9c123a0 Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'export-dev': Unable to import required dependencies:
numpy: cannot import name 'WinDLL' from 'ctypes' (/var/lang/lib/python3.7/ctypes/__init__.py)
END RequestId: e302cee0-3c51-453a-84c1-6eb1f9c123a0
REPORT RequestId: e302cee0-3c51-453a-84c1-6eb1f9c123a0  Duration: 1.65 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 70 MB  

I do not use the ctypes, WinDLL or any related packages explicitly in my code.

Answer 1

Aws lambda will throw you an error if you don't have the correct version of dependencies packaged with your code, which may depend on the OS (lambda runs on linux) and the python version.

Based on your requirements, it's pandas throwing you the error. To run pandas on lambda, you need to include the following packages:

pandas - code compiled for the linux, which is what lambda runs you. You can find it here https://pypi.org/project/pandas/#files download the 'manylinux' version of the .whl file, that matches your python lambda version.

• e.g. if you are running py3.7, then get pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl

• Unzip the contents of the .whl file into the root folder of your lambda folder. This is the library version that lambda needs

• Note for pandas 0.25+, you also need to include the pytz package as well, see note below on requests

numpy - You can now get in lambda (tested for py3.7) through installing a 'layer' through the lambda console, see screenshots below.

• Doesn't seem there's a layer for py3.8 yet so you'll need to do download the correct .whl file from pypi, just like pandas https://pypi.org/project/numpy/#files

Side note on requests

• Notice that the package here https://pypi.org/project/requests/#files only have a 'none-any' version, that means the source doesn't need to be compiled, so you can safely include the version you got from pip

• this applies to the pytz dependency of pandas as well

Screenshots installing layers in aws console

Answer 2

Since numpy is written in C you should build it for a linux distribution. I recommend you using the serverless framework because it will simplify your life a lot when you are using a Windows laptop.

Install the serverless framework and make sure you have docker

go to the root of your project and execute:

sls create --template aws-python

install the plugin for deploying python apps:

serverless plugin install -n serverless-python-requirements

in your serverless.yml file add:

plugins:
   - serverless-python-requirements

custom:
  pythonRequirements:
     dockerizePip: non-linux

make sure you adjust the path to your lambda function

functions:
  hello:
    handler: handler.hello

deploy with the correct libraries using

sls deploy