AWS Lambda Python Cryptography - Cannot open shared object files

Question

I am working on a Serverless Flask app that is deployed to AWS Lambda. The program uses the Cryptography library (using version 3.4.7). Locally, the program runs fine without any issue. However, whenever deployed on Lambda, the following error appears:

from cryptography.fernet import Fernet

File "/var/task/cryptography/fernet.py", line 16, in <module>

from cryptography.hazmat.primitives import hashes, padding

File "/var/task/cryptography/hazmat/primitives/padding.py", line 11, in <module>

from cryptography.hazmat.bindings._padding import lib

ImportError: /var/task/cryptography/hazmat/bindings/_padding.abi3.so: cannot open shared object file: No such file or directory

And when using some required functions from the "Hazardous Material" module, a very similar error appears:

File "/var/task/cryptography/hazmat/primitives/kdf/pbkdf2.py", line 28, in __init__
    backend = _get_backend(backend)
File "/var/task/cryptography/hazmat/backends/__init__.py", line 23, in _get_backend
    return default_backend()
File "/var/task/cryptography/hazmat/backends/__init__.py", line 14, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
File "/var/task/cryptography/hazmat/backends/openssl/__init__.py", line 6, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
File "/var/task/cryptography/hazmat/backends/openssl/backend.py", line 113, in <module>
    from cryptography.hazmat.bindings.openssl import binding
File "/var/task/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: /var/task/cryptography/hazmat/bindings/_openssl.abi3.so: cannot open shared object file: No such file or directory

However, the library files referenced do exist and they are in the exact paths indicated.

The app includes cryptography==3.4.7 in the requirements.txt as a dependency. Serverless then installs the packages while deploying to AWS with sls deploy. Serverless puts everything in a zip and uploads it to AWS. I can see all the files in this zip folder as expected.

I thought that it might be an issue with serverless incorrectly uploading or installing the packages when deploying, so I even tried including the cryptography folder directly in my project. However, despite any changes to the serverless configuration or the cryptography package itself, I have been unsuccessful in using this package on my deployed Lambda. Does anyone have any ideas what I could do to make this work?

Answer 1

One recommendation Amazon presents is to use the "sam" tool to build the distribution by using a Docker container. However, in my situation I wasn't able to use docker in the build environment.

Amazon provides some other documentation on how to use pip to install requirements by passing very explicit command line flags to ensure the Lambda environment's version is downloaded:

pip install \
  --platform manylinux2014_x86_64 \
  --implementation cp \
  --python 3.9 \
  --only-binary=:all: --upgrade \
  --target=build/package \
  cryptography==38.0.3

The "build/package" path will cause the dependencies to be downloaded and installed into that directory, to allow for easy zip for upload into a Lambda.

These flags can also be used if you have a setup.cfg or pyproject.toml file by using "." as the resource to load, rather than the explicitly named library.

I expect that as Amazon introduces new runtime environments and deprecates older ones, the given --platform and --python flags will need to change.

Answer 2

What I did to fix a similar problem, while trying to add a layer with the cryptography library to a lambda function, was to use the same runtime and processor architecture, in both the lambda function and layer.

For example, my problem was that I had a lambda function running with Python 3.9, and in a arm64 architecture. But I was creating the layer .zip file running python 3.8, and in a x86_64 architecture.

Don't ask me why it was easier for me to re-create the lambda in Python 3.8 and in x86_64, rather than the other way around.

Anyhow, as soon as I added the layer (with the cryptography library) to the lambda, it ran smoothly.

So, my theory is that you need to match both the runtime and architecture in order for a layer to work properly with a lambda function.

Additionally, now that I look up my solution, it is actually backed up by this article: https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html

(see the notes in it)