Pip pyproject.toml: Can optional dependency groups require other optional dependency groups?

Question

I am using the latest version of pip, 23.01. I have a pyproject.toml file with dependencies and optional dependency groups (aka "extras"). To avoid redundancies and make managing optional dependency groups easier, I would like to know how to have optional dependency groups require other optional dependency groups.

I have a pyproject.toml where the optional dependency groups have redundant overlaps in dependencies. I guess they could described as "hierarchical". It looks like this:

[project]
name = 'my-package'
dependencies = [
    'pandas',
    'numpy>=1.22.0',
    # ...
]

[project.optional-dependencies]
# development dependency groups
test = [
    'my-package[chem]',
    'pytest>=4.6',
    'pytest-cov',
    # ...
    # Redundant overlap with chem and torch dependencies
    'rdkit',
    # ...
    'torch>=1.9',
    # ...
]

# feature dependency groups
chem = [
    'rdkit',
    # ...
    # Redundant overlap with torch dependencies
    'torch>=1.9',
    # ...

]
torch = [
    'torch>=1.9',
    # ...
]

In the above example, pip install .[test] will include all of chem and torch groups' packages, and pip install .[chem] will include torch group's packages.

Removing overlaps and references from one group to another, a user can still get packages required for chem by doing pip install .[chem,torch], but I work with data scientists who may not realize immediately that the torch group is a requirement for the chem group, etc.

Therefore, I want a file that's something like this:

[project]
name = 'my-package'
dependencies = [
    'pandas',
    'numpy>=1.22.0',
    # ...
]

[project.optional-dependencies]
# development dependency groups
test = [
    'my-package[chem]',
    'pytest>=4.6',
    'pytest-cov',
    # ...
]

# feature dependency groups
chem = [
    'my-package[torch]',
    'rdkit',
    # ...
]
torch = [
    'torch>=1.9',
    # ...
]

This approach can't work because my-package is hosted in our private pip repository, so having'my-package[chem]' like the above example fetches the previously built version's chem group packages.

It appears that using Poetry and its pyproject.toml format/features can make this possible, but I would prefer not to switch our build system around too much. Is this possible with pip?

Answer 1

Maybe it's worth binding the 2 dependency groups into one, and see if that solves the problem, like shown in 'all'

[project]
name = "foo"
version = "0.1.0"

[project.optional-dependencies]
socks = ["pysocks"]
jwt = ["pyjwt"]
all = ["foo[socks,jwt]"]