pysftp library not working in AWS lambda layer

Question

I want to upload files to EC2 instance using pysftp library (Python script). So I have created small Python script which is using below line to connect

pysftp.Connection(
    host=Constants.MY_HOST_NAME,
    username=Constants.MY_EC2_INSTANCE_USERNAME,
    private_key="./mypemfilelocation.pem",
)
some code here .....
pysftp.put(file_to_be_upload, ec2_remote_file_path)

This script will upload files from my local Windows machine to EC2 instance using .pem file and it works correctly.

Now I want to do this action using AWS lambda with API Gateway functionality.

So I have uploaded Python script to AWS lambda. Now I am not sure how to use pysftp library in AWS lambda, so I found solution that add pysftp library Layer in AWS lambda Layer. I did it with

pip3 install pysftp -t ./library_folder

And I make zip of above folder and added in AWS lambda Layer.

But still I got so many errors like one by one :-

No module named 'pysftp'

No module named 'paramiko'

Undefined Symbol: PyInt_FromLong

cannot import name '_bcrypt' from partially initialized module 'bcrypt' (most likely due to a circular import)

cffi module not found

I just fade up of above errors I didn't find the proper solution. How can I can use pysftp library in my AWS lambda seamlessly?

Answer 1

I build pysftp layer and tested it on my lambda with python 3.8. Just to see import and basic print:

import json
import pysftp

def lambda_handler(event, context):
    # TODO implement
    print(dir(pysftp))
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

I used the following docker tool to build the pysftp layer:

• https://github.com/lambci/docker-lambda

So what I did for pysftp was:

# create pysftp fresh python 3.8 environment
python -m venv pysftp

# activate it
source pysftp/bin/activate

cd pysftp

# install pysftp in the environemnt
pip3 install pysftp  

# generate requirements.txt
pip freeze > requirements.txt

# use docker to construct the layer
docker run --rm -v `pwd`:/var/task:z lambci/lambda:build-python3.8 python3.8 -m pip --isolated install -t ./mylayer -r requirements.txt

zip -r pysftp-layer.zip .

And the rest is uploading the zip into s3, creating new layer in AWS console, setting Compatible runtime to python 3.8 and using it in my test lambda function.

You can also check here how to use this docker tool (the docker command I used is based on what is in that link).

Hope this helps