AWS Lambda python: .so module: ModuleNotFoundError: No module named 'regex._regex' when in subshell

Question

I'm trying to run some of my simple CI jobs (e.g: linters) into lambda functions for my python repositories.

I'm using gitlab-ci for this, and the basic architecture (not directly related to the problem but it may help the choices) is the following:

a CI jobs is defined as a set of shell commands in my case just running black --check . (a python linter): 1. the shell commands are sent to my lambda 2. the lambda git clone the repo 3. the lambda execute in a subprocess the commands 4. the result is returned

so my lambda looks like this:

import os
import stat
import sys
import json
import subprocess
import stat
import shutil

from git import Repo


def main(event, lambda_context):
    # I've commented out, so that if you like trying to reproduce it
    # at home, you don't need the additional dependencies :) 
    #Repo.clone_from("https://example.com/the/repo.git")

    # the command sent by the CI having a shebang in it
    # the quick and dirty solution we found is to write it into 
    # a shell script and then executing said script
    script = open("/tmp/script.sh", "w")
    script.write(event["command"])
    script.close()

    st = os.stat("/tmp/script.sh")
    os.chmod("/tmp/script.sh", st.st_mode | stat.S_IEXEC)

    # the copy is just to be extra-safe
    copy_env_variables = os.environ.copy()
    # for all the binary added by the requirements.txt
    # we edit PYTHONPATH for black in the subshell to find the modules
    lambda_root_dir = os.environ["LAMBDA_TASK_ROOT"]
    copy_env_variables["PATH"] = copy_env_variables["PATH"] + f":{lambda_root_dir}/bin"
    copy_env_variables["PYTHONPATH"] = (
        copy_env_variables.get("PYTHONPATH", "") + ":" + lambda_root_dir
    )
    proc = subprocess.Popen(
        "/tmp/script.sh",
        shell=False,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        env=copy_env_variables,
    )
    (stdout, _) = proc.communicate()

    return {"return_code": proc.returncode, "output": stdout.decode("utf-8")}

and the lambda is packaged with this script: (python3 is python3.7 and I'm on linux x86_64)

python3 -m pip install -r requirements.txt -t  ./
zip -r archive.zip . -x \*.pyc *.git*

the requirements.txt is the following:

gitpython
black

and when I execute the CI job, I got in the output of the subprocess:

$ black --check .
Traceback (most recent call last):
  File "/var/task/bin/black", line 6, in <module>
    from black import patched_main
  File "/var/task/black.py", line 15, in <module>
    import regex as re
  File "/var/task/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/var/task/regex/regex.py", line 402, in <module>
    import regex._regex_core as _regex_core
  File "/var/task/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ModuleNotFoundError: No module named 'regex._regex'

The special thing about regex._regexp is that it's a compiled module, named regex/_regex.cpython-37m-x86_64-linux-gnu.so , however if i import it directly in my lambda code, it works without problem.

Is there something I'm missing about the way python deal with modules coming from .so libraries ?

Answer 1

The problem come from the binary having /usr/bin/python3 in its shebang and pointing to python3.6, so python3.6 can't load .so compiled for python3.7

Strangely even though the Lambda was deployed with python3.7 runtime. In lambda python3.7 is located in /var/lang/bin/python3.7 , so for the moment I have modify my CI to symlink python3.7 in /var/lang/bin/python3.7 so the shebang generated by pip is the correct one for lambda,

Answer 2

You package lambda on your machine. Lambda runtime uses (probably) different linux distro. You may want to spin up amazonlinux:latest image with Docker and package lambda from there. (You may find the steps here https://smirnov-am.github.io/ci-ci-pipeline-for-aws-lambda-python-runtime/)