Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

requirements.txt - How to mark alternative packages

I am writing a project that relies on tensorflow, but that can be provided by either of two pip packages: tensorflow or tensorflow-gpu. My project works fine with either, but I don't want people running it on a machine without gpu support to have to install the extra overhead, but I still want people running on machines with gpu support to be able to leverage that. Is there a way to mark in my requirements.txt file that I require either tensorflow or tensorflow-gpu but not both?

EDIT:

In this specific case I should note that from the programmer's point of view, both tensorflow and tensorflow-gpu are identical, as they both provide a module tensorflow which has the same functions/classes/methods etc., and only differ in that tensorflow-gpu benefits from GPU acceleration. The problem that I am having is that if I put tensorflow in requirements.txt then in order to run with GPU acceleration, users would have to do pip install -r requirements.txt && pip uninstall tensorflow && pip install tensorflow-gpu which is not ideal, and if I instead put tensorflow-gpu in requirements.txt, then it will require a bunch of unnecessary system libraries (CUDNN etc) and wont work out-of-the-box for non-gpu users.

EDIT AGAIN

As a work-around, I've decided to provide two different requirement files, requirements.txt and requirements-gpu.txt, both of which include a shared -r .requirements-core.txt and add their respective version of tensorflow. That way people who want GPU support can pip install -r requirements-gpu.txt but the standard pip install -r requirements.txt will still work out-of-the box for everyone.

like image 328
phsyron Avatar asked Jun 05 '18 19:06

phsyron


1 Answers

You cannot condition download packages with requirements.txt, but you might do one of the following solutions:

1 - Install both packages tensorflow and tensorflow-gpu as dependencies and do a try / except to choose which package will actually be used, like:

tensorflow = null

try:
  tensorflow = __import__("tensorflow-gpu")
  tensorflow.operation_that_requires_gpu()
except:
  tensorflow = __import__("tensorflow")
enter code here

2 - On your project you ask the client to pass the dependency direct to you:

def my_function_that_uses_tensorflow(tensorflow):
  # do stuff

from my_module import my_function_that_uses_tensorflow
import tensorflow # or tensorflow = __import__("tensorflow-gpu")
my_function_that_uses_tensorflow(tensorflow)

3 - If tensorflow-gpu and tensorflow both install their package with the same tensorflow name on your site-packages, then my suggestion is do a try /except as I said on option number 1, but don't include tensorflow-gpu or tensorflow as a dependency of your package (treat it as a "peer dependency" that the code using your package should include as their dependency in order to use it):

try:
  import tensorflow
except:
  raise ImportError('You need to include tensorflow or tensorflow-gpu as a dependency in order to use this package')
like image 199
Guilherme D Heynemann Bruzzi Avatar answered Sep 18 '22 12:09

Guilherme D Heynemann Bruzzi