How to specify platform-specific dependences with Poetry?

Question

I want to constrain a single dependency (that is, a single package with a single name) based on the OS platform. For instance, the package version or the package origin (URL, local wheel, etc.) could change depending on the OS.

I tried the solution of linked in the documentation but that does not work. Poetry tries to install the wrong package for the wrong OS platform. I also searched StackOverflow and found 1 related question but it does not help.

As a practical use case, I want to install PyTorch 2.0.1 from PyPI on macOS and a specific wheel (with a specific version of CUDA) on Ubuntu. As such, my package specification is:

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    {platform = "linux", url = "https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl"},
    {platform = "darwin", version = "2.0.1"},
]

Unfortunately, on macOS Poetry tried to install the Linux package as mentioned in the error message:

Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing torch (2.0.1+cu118 https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl): Failed

  RuntimeError

  Package https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl cannot be installed in the current environment {'implementation_name': 'cpython', 'implementation_version': '3.10.11', 'os_name': 'posix', 'platform_machine': 'arm64', 'platform_release': '22.5.0', 'platform_system': 'Darwin', 'platform_version': 'Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000', 'python_full_version': '3.10.11', 'platform_python_implementation': 'CPython', 'python_version': '3.10', 'sys_platform': 'darwin', 'version_info': [3, 10, 11, 'final', 0], 'interpreter_name': 'cp', 'interpreter_version': '3_10'}

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:788 in _download_link
      784│             # Since we previously downloaded an archive, we now should have
      785│             # something cached that we can use here. The only case in which
      786│             # archive is None is if the original archive is not valid for the
      787│             # current environment.
    → 788│             raise RuntimeError(
      789│                 f"Package {link.url} cannot be installed in the current environment"
      790│                 f" {self._env.marker_env}"
      791│             )
      792│ 

Please not that I made sure that the lock file is consistent with pyproject.toml by running poetry lock before.

Is there a solution to this issue?

Answer 1

I found a solution that is not entirely satisfactory. This works in my specific case but could not be applicable to other situations. It consists in specifying the exact and explicit wheel URL for each OS platform instead on relying on PyPI for macOS:

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    {platform = "darwin", url = "https://download.pytorch.org/whl/cpu/torch-2.0.1-cp310-none-macosx_11_0_arm64.whl"},
    {platform = "linux", url = "https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl"},
]

---

Note that if what you are looking for is a multi-platform environment for PyTorch using Poetry working on Windows + CUDA, Ubuntu + CUDA and macOS + MPS (but without downloading the CUDA wheel) out of the box, then this is what your TOML file should look like:

[tool.poetry]
name = "my-pytorch-project"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.10,<3.13"
numpy = "^1.26.3"
torch = [
    { version = "^2.0.0", source = "PyPI", platform = "darwin" },
    { version = "^2.0.0", source = "pytorch", platform = "!=darwin" },
]
torchvision = [
    { version = "^0.15.0", source = "PyPI", platform = "darwin" },
    { version = "^0.15.0", source = "pytorch", platform = "!=darwin" },
]

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

This snippet is using an explicit dependency on the CUDA wheels for Windows and Linux (inaccurately !darwin) and uses the CPU wheels (that supports MPS) for macOS.