Prevent pip from installing some dependencies

Question

We're developing an AWS Lambda function for Alexa skill in Python and using pip to install the ask-sdk package to our dist/ directory:

pip install -t dist/ ask-sdk

The trouble is with the -t dist/ because pip wants to have all the dependencies there, even if they are installed system-wide.

Now, ask-sdk has a dependency on boto3 which pulls in a whole lot of other packages. However the AWS Lambda runtime environment provides boto3 and there is no need to package that and its dependencies with our code. I do have boto3 installed in the system and import boto3 works, so I thought pip should be happy, but because of -t dist/ it always installs it.

Can I somehow install just ask-sdk and its dependencies that don't exist in the system, e.g. ask-sdk-core, but not those that are already installed?

One way is to list all the modules and use pip --no-deps but that means a constantly keeping track of the dependencies manually, we would like to avoid that.

Somehow I would like to tell pip: if the package is already installed, even if not in -t dist/ don't put a copy in dist/.

Is that possible?

Thanks!

Answer 1

You can try the option

  --no-dependencies

To ignore all dependencies.

To exclude specific, you can put it in requirements file and pass it:

pip install --no-deps -r requirements.txt

Answer 2

Although you can't tell pip to "install all dependencies except those required by boto3", you can generate the needed requirements.txt by computing the difference between boto3 and ask-sdk from pip freeze output (tested with Python 3.6.6):

# get boto3 requirements
pip install boto3 -t py_lib.boto3
PYTHONPATH=py_lib.boto3 pip freeze > requirements-boto3.txt

# get ask-sdk requirements
pip install ask-sdk -t py_lib.ask-sdk
PYTHONPATH=py_lib.ask-sdk pip freeze > requirements-ask-sdk.txt

# compute their difference
grep -v -x -f requirements-boto3.txt requirements-ask-sdk.txt > requirements-final.txt

# patch to add one missing dep from boto3
# aws don't have this for some reason
grep urllib3 requirements-boto3.txt >> requirements-final.txt

The requirements-final.txt contains the following:

ask-sdk==1.5.0
ask-sdk-core==1.5.0
ask-sdk-dynamodb-persistence-adapter==1.5.0
ask-sdk-model==1.6.2
ask-sdk-runtime==1.5.0
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1

To install the final set of dependencies to a folder:

pip install --no-deps -r requirements-final.txt -t py_lib

By skipping the boto3 dependencies, you can save about 45M of data from your python dependencies. The ask-sdk dependencies are only about 7.5M (2.1M compressed), allow you to use the build-in lambda code editor if you need to.