How to run python code on AWS lambda with package dependencies >500MB?

Question

The requirement is that I have to trigger a SageMaker endpoint on lambda to get predictions(which is easy) but have to do some extra processing for variable importance using packages such as XGBoost and SHAP.

I am able to hit the endpoint and get variable importance using the SageMaker Jupyter notebook. Now, I want to replicate the same thing on AWS lambda.

1) How to run python code on AWS lambda with package dependencies for Pandas, XGBoost and SHAP (total package size greater than 500MB). The unzipped deployment package size is greater than 250 MB, hence lambda is not allowing to deploy. I even tried using lambda function from Cloud9 and got the same error due to size restrictions. I have also tried lambda layers, but no luck.

2) Is there a way for me to run the code with such big packages on or through lambda bypassing the deployment package size limitation of 250 MB

3) Is there a way to trigger a SageMaker notebook execution through lambda which would do the calculations and return the output back to lambda?

Answer 1

Try to upload your dependencies to the Lambda Layer. FYI: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

Answer 2

In addition to use multiple layers for your dependencies - you may want to reduce the *.so files by linux strip command which discards symbols from compiled object files which may not necessary in production

In order to strip all *.so -

1. use linux/docker container with access to your dependencies directory
2. cd to your dependencies directory
3. Run

find . -name *.so -exec strip {} \;

Will execute strip command on every *.so file in the current working directory recursively.

It helped me reduce one of my dependencies objects from 94MB to just 7MB