Reduce python AWS Lambda size for deployment on CloudFront

Question

We have a Lambda function in python using below libraries

pyOpenSSL==18.0.0

protobuf==3.14.0

The python code is less than 100KB, but with the above dependencies included the size goes to more than 4MB.

Requirement is to deploy this Lambda on a CloudFront distribution (Lambda@Edge) for which size of function with dependencies cannot exceed 1MB.

Is there any way to achieve this?

Any help is appreciated. Any help is appreciated.

Answer 1

You'll probably have to modify your deployment tooling a lot.

If these libraries have any compiles binaries, you can remove the debug symbols from them with the strip command (which modifies the files in place)

Something like:

find ./ -name '*.so' -type -f -exec strip "{}" \;

You can also try the same by for *.so*, since there are other files that end in .so.400. Although I have seen cases where this broke the files. So ensure you test well.

Have a look at the size of those libraries. (e.g. pip install pyOpenSSL --target test). Look at what's inside. Look at which folders are the biggest. GUI apps like this make that easy. If you're on a terminal, you may want something like du -s ./ | sort -n.

If there are folders and files that are only for unit testing, try deleting them. (e.g. any folders called test or tests.) Similarly, you can delete README.* files, or other documentation.

I don't know about Lambda@Edge, but for normal lambda, the AWS SDKs are already installed. So if your dependencies depend on the AWS SDKs, you're doubling up, and can delete them. I doubt these ones do though.

An extreme workaround is to zip up your dependencies separately to your main code. Then when the lambda runs, the first thing you do is download the zip from S3, unzip it, and then import those libraries. If Lambda@Edge is like normal lambda, that download will be cached within each Lambda VM. So for a frequently used Lambda, that won't cost anything on warm starts.