setup.py with dependecies installed by conda (not pip)

Question

I am working on an existing Python 3 code-base that provides a setup.py so the code is installed as a Python library. I am trying to get this internal library installed with its own dependencies (the usual data science ones e.g. pandas, pyodbc, sqlalchemy etc).

I would like to have this internal library to deal with these dependencies and assume that if that library is installed, then all the transitive dependencies are assumed to be installed. I also would like to have the Anaconda (conda) version of the package rather than the pip version.

I started with a requirements.txt, but moved quickly to this field in setup.py:

  install_requires=[
      "pyodbc>=4.0.27",
      "sqlalchemy>=1.3.8",
      "pandas>=0.25.1",
      "requests>=2.22.0",
      "assertpy>=0.14",
      "cycler>=0.10.0",
  ]

However when I run the installation process:

• either with python setup.py install --record installed_files.txt
• or with pip install .

I see that there is some gcc / C++ compilation going on that shows logs about Python wheels (I don't completely understand the implications of Python eggs and Python wheels, but AFAIK if conda is available then I should go with the conda version rather than eggs/wheels because then I don't have to take care of the C++ code underneath the Python code).

I really would prefer having conda to install these C++ blobs wrapped in some Python code as a libraries e.g. pandas.

• is it possible at all to have conda driving the installation process described in setup.py so I am not dealing with gcc?
• how can I make sure that other Python code depending on this internal library (installed via setup.py) is using the same (transitive) dependencies defined in that setup.py?

Regardless the installation method, how can I make sure that the dependencies for e.g. pandas are installed as well? Sometimes I see that numpy as a dependency of pandas is not installed when running setup.py, but I would like to avoid doing this manually (e.g. with some requirements.txt file).

Answer 1

pip doesn't know about conda, so you cannot build a pip-installable package that pulls in its dependencies from conda channels.

conda doesn't care about setup.py, it uses a different format for recording dependencies.

To install your code with conda, you should create a conda package, and specify your dependencies in a meta.yaml file. Refer to the documentation of "conda build" for details.

https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html