Unable to import lxml etree on aws lambda

Question

{
  "errorMessage": "Unable to import module 'lambda_function': 
        cannot import name   'etree' from 'lxml' (/var/task/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

Also tried https://gist.github.com/allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639:

{
  "errorMessage": "Unable to import module 'lambda_function': 
     cannot import name 'etree' from 'lxml' 
     (/var/task/lxml-4.3.4-py3.6-linux-x86_64.egg/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

I am running on Ubuntu 18.04 on my local machine, and have also tried using the "Amazon Linux" image on an ec2 instance to build the bundle.

I have also tried, while within the activated venv:

STATIC_DEPS=true pip3 install lxml --target ./package --upgrade --no-cache-dir

I have also tried copying the shared object files based on pulling which files were opened when running the script with strace:

#! /bin/bash

export Z=$(pwd)/ok-daily-lambda.zip
rm $Z
zip $Z lambda_function.py
zip $Z __init__.py

for dir in $(find venv_here/lib/python3.6/site-packages)
do
    if [ -d $dir ] ; then
        pushd $dir
        echo zip -r9 $Z $(pwd)
        zip -r9 $Z $(pwd)
        popd
    fi
done

export LIBD=$(pwd)/lib
mkdir -p $LIBD

cp "/home/jmunsch/.local/lib/python3.6/site-packages/.libs_cffi_backend/libffi-d78936b1.so.6.0.4" $LIBD
cp "/lib/x86_64-linux-gnu/libbz2.so.1.0" $LIBD
cp "/lib/x86_64-linux-gnu/libc.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libdl.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libexpat.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libgcc_s.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/liblzma.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libm.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_dns.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_files.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libpthread.so.0" $LIBD
cp "/lib/x86_64-linux-gnu/libresolv.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/librt.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libtinfo.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libudev.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libutil.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libuuid.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libz.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libffi.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/liblz4.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libmpdec.so.2" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libssl.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libstdc++.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libzstd.so.1" $LIBD

zip -r $Z $LIBD

AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="xxx" AWS_DEFAULT_REGION="us-east-1" aws lambda update-function-code --function-name ok-today --zip-file fileb://ok-daily-lambda.zip

Here is the directory structure of the most recent zip file:

• https://gist.github.com/allen-munsch/5ddc27873e64db6ff106ab828febf434

Related:

• AWS Lambda not importing LXML
• LXML: Cannot import etree
• https://lxml.de/installation.html
• https://lxml.de/build.html
• lxml cannot import element etree
• Shared Libraries in Java AWS Lambda Project
• https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Answer 1

I came across a similar problem and I figured out one quick workaround

Using pre-compiled build of lxml

Download https://github.com/shubh2502/aws-lambda-lxml

1. Folder 3.6.4 and 3.8.0 are lxml versions

2. Inside lxml there are two builds python27 and python36

3. As per AWS Lambda python version choose either one of them

4. Inside python27 and python36 there is lxml folder

5. Zip code with lxml folder and make sure python has the same version

6. In Case of AWS Lambda layer put lxml folder into this structure -

   python/lib/python3.6/site-packages

I spent lots of time in docker and building these stuff, this method was savior for me, I hope this will help you out

Answer 2

Reason:

The .so files compiled on a different OS. (In my case macOS.)

Solution:

The following article was very helpful:

How do I create a Lambda layer using a simulated Lambda environment with Docker?

All I had to do is:

cd /path_to_requirements_txt/

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.7" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.7/site-packages/; exit"

It created python/lib/python3.7/site-packages/ with all the dependencies!