How do I install pip packages through a conda-build recipe?

Question

I am trying to make a conda package out of a data pipeline. The idea is to distribute the package locally through our department for quick deployment.

Our pipeline uses 2 packages that are not available for Windows in common conda channels: SimpleITK and scitools3. We typically resort to installing these 2 packages through pip after we have created our conda environment with all other conda package dependencies.

I have made the following conda package recipe meta.yaml:

{% set name = "package_name" %}
{% set version = "1.0.0" %}

package:
  name: "{{ name|lower }}"
  version: "{{ version }}"

source:
  path: ..

build:
  number: 0
  script: "{{ PYTHON }} -m pip install . --no-deps --ignore-installed -vvv"
  entry_points:
    - cmr = package_name.api.cli.command_line_interface_segmentation:main
    - cupdate = package_name.api.cli.continuous_update:parse_and_run

requirements:
  host:
    - python
    - pip
  run:
    - python=3.7.4
    - tensorflow-gpu=2.1
    - pyqt
    - opencv=4.1 # conda install -c conda-forge opencv
    - pyyaml
    - scikit-image
    - scikit-learn
    - tqdm
    - deprecated
    - tabulate
    - pandas
    - xlsxwriter
    - h5py=2.10
    - psutil # for memory profiling
    - pympler # form memory profiling
#    - SimpleITK==1.2.4  # When this is commented, the package builds
    - xnat==0.3.18
#    - scitools3>=1.0  # When this is commented, the package builds

test:
  imports:
    - package_name

about:
  summary: 'Repository for processing data and training models.'

Our repository structure looks like following:

/package-name
    /conda.recipe
        bld.bat
        build.sh
        meta.yaml
    /package_name
        /api
            /cli
                command_line_interface_segmentation.py
                continuous_update.py
        ...
        /other_package_code
    requirements.txt
    setup.py

I then build the conda package through my terminal:

conda-build package-name

My question is, is it possible to install pip package dependencies through the conda recipe?

I've tried reading up all the official docs, example recipes from conda-forge, as well as questions on various sites. I've mostly come across people mentioning this would be a nice feature to have, but that there is no direct support for pip dependencies. However, I have not come across any information indicating that this can't be done.

My first idea:

I noticed there is a script parameter under build in meta.yaml. From what I've seen, most recipes tend to include something along the lines of:

build:
    script: "{{ PYTHON }} -m pip install . --no-deps --ignore-installed"

I don't have a complete grasp on what this script call is doing. Could pip packages be somehow installed through this script parameter? I have tried different variation of the following with no success:

script: "{{ PYTHON }} -m pip install SimpleITK scitools3"

Second idea:

Could the bld.bat or build.sh be modified to automatically install pip packages? I am not so knowledgeable when it comes to making shell scripts or when specifically these files get called during the conda-build process. All I know is that bld.bat is called in Windows and build.sh is called in MacOS and Linux.

Third idea:

From what I can see in the bld.bat and build.sh files, the setup.py file is called. For example in bld.bat:

"%PYTHON%" setup.py install
if errorlevel 1 exit 1

When and why does the setup.py file get called at any point during the conda-build call? Do the requirements listed in setup.py get installed at any point alongside/into the conda package I am creating?. Another option could also be to call setup.py through the meta.yaml script parameter.

Again, my understanding of how conda packages are created is quite shallow. Any pointers on how to better understand conda package-making would be greatly appreciated.

Answer 1

One should not do that. What OP proposes would essentially undermine the Conda package management system. Imagine that you did manage to do this and then later someone creates a Conda version of the same package. Now your package is silently going to clobber what Conda is trying to explicitly manage, resulting in a potentially unstable or at least desynchronized state.

If you wish to build a package that has a dependency that is not yet part of the Conda system, you need to first convert the non-Conda package to a Conda package. For packages in PyPI, this is vastly simplified by conda skeleton pypi, and the common practice is to create a Conda Forge recipe for the package. This is now further simplified by the grayskull tool.

For the SimpleITK package, that already has a Conda Forge feedstock, just that they aren't building for Windows. This could be a simple matter of the feedstock maintainers not working on Windows, in which case, they might be more than happy to add you or your colleagues on as co-maintainers to handle that part. Consider dropping them an issue or contacting the maintainers directly about your interest in a win-64 build.