Python poetry - how to install optional dependencies?

Question

Python's poetry dependency manager allows specifying optional dependencies via command:

$ poetry add --optional redis

Which results in this configuration:

[tool.poetry.dependencies]
python = "^3.8"
redis = {version="^3.4.1", optional=true}

However how do you actually install them? Docs seem to hint to:

$ poetry install -E redis

but that just throws and error:

Installing dependencies from lock file

[ValueError]
Extra [redis] is not specified.

Answer 1

You need to add a tool.poetry.extras group to your pyproject.toml if you want to use the -E flag during install, as described in this section of the docs:

[tool.poetry.extras]
caching = ["redis"]

---

The key refers to the word that you use with poetry install -E, and the value is a list of packages that were marked as --optional when they were added. There currently is no support for making optional packages part of a specific group during their addition, so you have to maintain this section in your pyproject.toml file by hand.

The reason behind this additional layer of abstraction is that extra-installs usually refer to some optional functionality (in this case caching) that is enabled through the installation of one or more dependencies (in this case just redis). poetry simply mimics setuptools' definition of extra-installs here, which might explain why it's so sparingly documented.

Answer 2

I will add that not only you have to have this extras section added by hand, as well your optional dependencies cannot be in dev section.

Example of code that won't work:

[tool.poetry]
name = "yolo"
version = "1.0.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "2.7"
Django = "*"

[tool.poetry.dev-dependencies]
pytest = "*"
ipdb = {version = "*", optional = true}

[tool.poetry.extras]
dev_tools = ["ipdb"]

But this WILL work:

[tool.poetry]
name = "yolo"
version = "1.0.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "2.7"
Django = "*"
ipdb = {version = "*", optional = true}

[tool.poetry.dev-dependencies]
pytest = "*"

[tool.poetry.extras]
dev_tools = ["ipdb"]

Answer 3

Up-voted Drachenfels's answer.

Dev dependency could not be optional, otherwise, no matter how you tweak it with extras or retry with poetry install -E, it will just never get installed.

This sounds like a bug but somehow by design,

...this is not something I want to add. Extras will be referenced in the distributions metadata when packaging the project but development dependencies do not which will lead to a broken extras.

— concluded in Poetry PR#606 comment by one maintainer. See here for detailed context: https://github.com/python-poetry/poetry/pull/606#issuecomment-437943927

---

I would say that I can accept the fact that optional dev-dependency cannot be implemented. However, at least Poetry should warn me when I have such a config. If so, I wouldn't have been confused for a long time, reading each corner of the help manual and found nothing helpful.

---

I found some people did get trap in this problem (Is poetry ignoring extras or pyproject.toml is misconfigured?) but their questions are closed, marked duplicated and re-linked to this question. Thus I decided to answer here and give more details about this problem.