Pip install Python package within AWS Lambda?

Question

I'm trying to pip install a package in an AWS Lambda function.

The method recommended by Amazon is to create a zipped deployment package that includes the dependencies and python function all together (as described in AWS Lambda Deployment Package in Python). However, this results in not being able to edit the Lambda function using inline code editing within the AWS Lambda GUI.

So instead, I would like to pip install the package within the AWS Lambda function itself. In AWS Lambda, the filesystem is read-only apart from the /tmp/ directory, so I am trying to pip install to the /tmp/ directory. The function is only called once-daily, so I don't mind about the few extra seconds required to re-pip install the package every time the function is run.

My attempt

def lambda_handler(event, context):
    # pip install dependencies
    print('begin lambda handler')
    import subprocess
    import sys
    subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split())
    from cryptography.fernet import Fernet
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)
    print('end lambda handler')

However, this results in the error

[ERROR] ModuleNotFoundError: No module named 'cryptography'

I have also tried replacing the subprocess call with the following, as recommended in this stackoverflow answer

    cmd = sys.executable+' -m pip install cryptography -t dependencies --no-cache-dir'
    subprocess.check_call(cmd.split())

However, this results in the error

OSError: [Errno 30] Read-only file system: '/var/task/dependencies'

Answer 1

I solved this with a one-line adjustment to the original attempt. You just need to add /tmp/ to sys.path so that Python knows to search /tmp/ for the module. All you need to do is add the line sys.path.insert(1, '/tmp/').

Solution

import os
import sys
import subprocess

# pip install custom package to /tmp/ and add to path
subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sys.path.insert(1, '/tmp/')
from cryptography.fernet import Fernet

def lambda_handler(event, context):
    # pip install dependencies
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)

Output

Hello stackoverflow!

Note - as @JohnRotenstein mentioned in the comments, the preferred method to add Python packages is to package dependencies in an AWS Lambda Layer. My solution just shows that it is possible to pip install packages directly in an AWS Lambda function.