aws lambda - failed to find libmagic

Question

I'm using in my lambda function the magic library to determine the file`s type. I first deployed it to a local container to check that everything works.

My DockerFile :

FROM lambci/lambda:build-python3.8
WORKDIR /app
RUN mkdir -p .aws
COPY requirements.txt ./
COPY credentials /app/.aws/
RUN mv /app/.aws/ ~/.aws/
RUN pip install --no-cache-dir  -r requirements.txt
RUN pip install --no-cache-dir  -r requirements.txt -t "/app/dependencies/"
WORKDIR /app/dependencies
RUN zip -r  lambda.zip *

requirements.txt :

python-magic
libmagic

In my local container when I run tests on the lambda logic everything went ok and passed (including the part that uses the magic code..).

I created a zip that contains the lambda.py code and with the python dependencies (last 3 lines in the docker file). When I upload the zip to aws and test the lambda I'm getting the following error :

{
  "errorMessage": "Unable to import module 'lambda': failed to find libmagic.  Check your installation",
  "errorType": "Runtime.ImportModuleError"
}

As you can see, on my local container I'm using baseline image lambci/lambda:build-python3.8 that should be the same aws uses when the lambda is launching.

I tried also to add python-magic-bin==0.4.14 to the requirements.txt instead of the magic and libmagic but it didnt help either because it seems that this module is for windows.

Into the lambda.zip I put also the lambda.py which is the file that includes my lambda function :

import boto3
import urllib.parse
from io import BytesIO
import magic

def lambda_handler(event, context):
    s3 = boto3.client("s3")
    if event:
        print("Event : ", event)
        event_data = event["Records"][0]
        file_name = urllib.parse.unquote_plus(event_data['s3']['object']['key'])
        print("getting file: {}".format(file_name))
        bucket_name = event_data['s3']['bucket']['name']
        file_from_s3 = s3.get_object(Bucket=bucket_name, Key=file_name)
        file_obj = BytesIO(file_from_s3['Body'].read())
        print(magic.from_buffer(file_obj.read(2048)))

What am I doing wrong ?

Answer 1

While using filetype as suggested by other answers is much simpler, that library does not detect as many file types as magic does.

You can make python-magic work on aws lambda with python3.8 by doing the following:

• Add libmagic.so.1 to a lib folder at the root of the lambda package. This lib folder will be automatically added to LD_LIBRARY_PATH on aws lambda. This library can be found in /usr/lib64/libmagic.so.1 on an amazon linux ec2 instance for example.
• Create a magic file or take the one available on an amazon linux ec2 instance in /usr/share/misc/magic and add it to your lambda package.
• The Magic constructor from python-magic takes a magic_file argument. Make this point to your magic file. You can then create the magic object with magic.Magic(magic_file='path_to_your_magic_file') and then call any function from python-magic you like on that object.

These steps are not necessary on the python3.7 runtime as those libraries are already present in aws lambda.