How does Poetry work regarding binary dependencies? (esp. numpy)

Question

Until now I have been using conda as virtual environment and dependency management. However, some stuff does not work as expected when transfering my environment.yml file from my development machine to the production server. Now, I would like to look into alternatives. Poetry seems nice, especially because

poetry also maintains a lock file, and it has a benefit over pipenv because it keeps track of which packages are subdependencies. (https://realpython.com/effective-python-environment/#poetry)

which might improve stability quite a bit. However, I am working on science-heavy projects (matrices, data science, machine learning), so in practise I need the scipy stack (e.g. numpy, pandas, scitkit-learn).

Python became too slow for some pure computational workloads so numpy and scipy were born. [...] They are written in C and just wrapped as a python library.

Compiling such libraries brings a set of challenges since they (more or less) have to be compiled on your machine for maximum performance and proper linking with libraries like glibc.

Conda was introduced as an all-in-one solution to manage python environments for the scientific community.

[...] Instead of using a fragile process of compiling libraries on your machine, libraries are precompiled and just downloaded when you request them. Unfortunately, the solution comes with a caveat - conda does not use PyPI, the most popular index of python packages.

(https://modelpredict.com/python-dependency-management-tools#fnref:conda-compiling-challenges)

As far as I know, this doesn't even do Conda justice, because it does quite a bit of optimization to get the most out of my CPU/GPU/architecture for numpy. (https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/#Myth-#6:-Now-that-pip-uses-wheels,-conda-is-no-longer-necessary)

https://numpy.org/install/ itself advises to use conda, but also says that one can install via pip (and poetry uses pypi)

For users who know, from personal preference or reading about the main differences between conda and pip below, they prefer a pip/PyPI-based solution, we recommend:

[...] Use Poetry as the most well-maintained tool that provides a dependency resolver and environment management capabilities in a similar fashion as conda does.

I would like to get the stability of the poetry setup and the speed of the conda setup.

How does poetry handle binary dependencies? Does it also, like conda, consider my hardware?

If poetry not deliver in this regard, can I combine it with conda?

Answer 1

numpy provides several wheel files for different os, cpu architecture and python versions. wheel packages are precompiled, so the target system doesn't have to compile the package.

poetry is able to choose the right wheel for you, depending on your system.

Saying this, I would recommend using poetry, as long as you just need python packages, which are also available at pypi. As soon as you need other, non-python tools, stick to conda. (Disclaimer: I'm one of the maintainer of poetry).

Also related: https://github.com/python-poetry/poetry/issues/1904

Answer 2

On MAC Os BigSur (11.1)

I've being facing issues all day with numpy failing install (with pandas) for Python 3.9.1.6 and poetry 1.1.4.

I understood that poetry uses pip which does not use the wheel version of numpy when pip <= 20.2.x. And fresh installation of Python 3.9.1.6 does not upgrade pip !

This is how I managed to install what I needed :

First check the default version of Python linked to poetry (reinstall it if it does not fit)

poetry env info | grep -i python
Python:         3.9.1
Implementation: CPython
Python:   /usr/local/Cellar/[email protected]/3.9.1_6/Frameworks/Python.framework/Versions/3.9

Install you project and run poetry init without any dependencies you seek (you'll use poetry add <deps> juste after...)

You should obtain something similar to this pyproject.toml :

[tool.poetry]
name = "project1"
version = "0.1.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Launch the poetry shell

➜  project1 poetry shell

Python 2.7 will no longer be supported in the next feature release of Poetry (1.2).
You should consider updating your Python version to a supported one.

Note that you will still be able to manage Python 2.7 projects by using the env command.
See https://python-poetry.org/docs/managing-environments/ for more information.

Spawning shell within /Users/vincent/Library/Caches/pypoetry/virtualenvs/project1-5XOg8Qie-py3.9
➜  project1 . /Users/vincent/Library/Caches/pypoetry/virtualenvs/project1-5XOg8Qie-py3.9/bin/activate
(project1-5XOg8Qie-py3.9) ➜  project1

At this point upgrade pip, wheel and setuptools !

(project1-5XOg8Qie-py3.9) ➜  project1 $ pip install --upgrade pip wheel setuptools
Looking in indexes: https://pypi.python.org/simple/
Collecting pip
  Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
Collecting wheel
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting setuptools
  Using cached setuptools-51.1.2-py3-none-any.whl (784 kB)
Installing collected packages: pip, wheel, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.2.4
    Uninstalling pip-20.2.4:
      Successfully uninstalled pip-20.2.4
  Attempting uninstall: wheel
    Found existing installation: wheel 0.35.1
    Uninstalling wheel-0.35.1:
      Successfully uninstalled wheel-0.35.1
  Attempting uninstall: setuptools
    Found existing installation: setuptools 50.3.2
    Uninstalling setuptools-50.3.2:
      Successfully uninstalled setuptools-50.3.2
Successfully installed pip-20.3.3 setuptools-51.1.2 wheel-0.36.2

Now installation should be ok

(project1-5XOg8Qie-py3.9) ➜  project1 poetry install
Installing dependencies from lock file

Package operations: 5 installs, 0 updates, 0 removals

  - Installing six (1.15.0)
  - Installing numpy (1.19.5)
  - Installing python-dateutil (2.8.1)
  - Installing pytz (2020.5)
  - Installing pandas (1.2.0)

Installing the current project: project1 (0.1.0)

Yeepee !