poetry build with common library

Question

I have a repository with multiple Python projects that use Poetry for dependency management. The directory structure is:

• proj1

  • src/
  • pyproject.toml

• proj2

  • src/
  • pyproject.toml

• common_proj

  • src/
  • pyproject.toml

The proj1 and proj2 projects both depend on common_proj. Currently, I'm building proj1 and proj2 like this:

cd proj1
cp -r ../common_proj/src ./common_proj
poetry build

Then I use the generated wheel file in production.

The pyproject.toml files for proj1 and proj2 include common_proj like this:

packages = [
    {include = "proj2"},
    {include = "common_proj" },
]

What is the correct way to manage the common_proj dependency with Poetry, so that its dependencies are installed properly?"

Answer 1

I have recently been dealing with a similar problem to this. I have a couple of solutions to this and myself have implemented some of these for different problems depending on the use case.

1. Publish common_proj as its own package

If you publish common_proj as its own package you can then install this in your other packages. I believe this is probably the nicest approach. It also decouples the different packages a little more which in the long run might be of much more value.

2. Use a script for building and publishing

This is a bit more involved than option 1. However, I have found that this one works for cases where I am told I am absolutely not allowed to publish the common_proj as a package. The script here is pretty much exactly as you described above. In usecases where you have multiple different common_proj and some subset of them that might be used then it can get more complicated.

3. The docker approach

This is a similar approach to the one above. However, I find it works slightly nicer assuming you will already be packaging the projects up into dockerfiles. I have used this a couple of times where all the projects get deployed to serverless functions. With them all using similar classes that sit in the shared folder.

Rather than having multiple different projects all the code is in 1 project as a mono-repo.

project
-- proj_1
   -- files.py
-- proj_2
   -- files.py
-- common_proj
   -- files.py
pyproject.toml
poetry.lock

The dockerfile can be build based on which project you are trying to build. You then have a script you run that will read the pyproject.toml and remove the libraries for projects not currently being built. This works best when the projects are very similar often with the same required packages but need to be deployed separately.

Conclusion

Publishing the common_proj as a package is the nicest, it makes life the easiest. The other solutions have trade-offs and rely on adding extra complexity to your projects to manage packages. That said this can all be automated with scripts that you run instead of those you are used to. I would really recommend against them unless you absolutely cannot publish the package.

Edited 2023-07-09 to make it clear when the docker solution makes sense.