How to do in order to package python modules separately when organized in the same project?

Question

I am more experienced in Java and Maven and a real beginner in Python, so I don't really have an idea about what possible to do and what not.

In Java, using Maven we can manage dependencies versions in a file (a pom parent / the aggregator module), then re-declare in pom.xml of the sub-modules the necessary dependencies while omitting their version being managed by the pom parent. This way, we can distribute the sub-modules physically separately (Ex: every module in a machine) and every module would package only its required dependencies, the libraries that are defined in its own pom file.

Now back to Python, currently I use a setup.py file which, to my understanding, should be in the root module. But then it packages the whole modules as one library.

Is there a way to manage better the modules and their dependencies and be able to deploy every (chosen) module separately as an independent package with its own needed libraires? I want to be able to choose the modules to package separately, I don't want that every folder containing __init__.py to be packaged seprately.

Currently I use Pipenv to manage my dependencies, but I am ready to drop it if it doesn't satisfy the design I have explained above.

Thank you

Answer 1

requirements.txt

In java, as you said, we have our pom.xml

In python you have: requirements.txt with content like this:

# Requirements without Version Specifiers #`
nose
nose-cov
beautifulsoup4

# Requirements with Version Specifiers #`
docopt == 0.6.1             # Version Matching. Must be version 0.6.1
keyring >= 4.1.1            # Minimum version 4.1.1
coverage != 3.5             # Version Exclusion. Anything except version 3.5
Mopidy-Dirble ~= 1.1        # Compatible release. Same as >= 1.1, == 1.*

To install, run this:

pip install -r requirements.txt

---

dependencies in java/maven

With maven we have a very good dependency management:

org.acme.demo.springboot:acme-api:jar:1.0.0
+- mysql:mysql-connector-java:jar:8.0.13:compile
+- io.jsonwebtoken-jjwt:jar:0.9.1:compile
|  \- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
|     +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
|     \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
+- com.jayway.jsonpath:json-path:jar:2.4.0:compile
|  +- net.minidev:json-smart:jar:2.3:compile
|  |  \- net.minidev:accessors-smart:jar:1.2:compile
|  |     \- org.ow2.asm:asm:jar:5.0.4:compile
|  \- org.slf4j:slf4j-api:jar:1.7.25:compile

For example: My app is acme-api and has these dependencies in its pom.xml:

• mysql:mysql-connector-java which depends of nothing
• io.jsonwebtoken-jjwt wich depends of jackson-databind, jackson-annotations and jackson-core

If you look the jackson-core source code, you will find another pom.xml with just the dependencies required by jackson-core

So, in java/maven any source code repository must have the pom.xml (app or library) in which we can find or download the required libraries

---

dependencies in python

The previous strategy in java/maven related to pom.xml in any source code, is not used in python.

I reviewed a couple of public libraries and I don't find the requirements.txt inside of them o_O

• https://github.com/jupyter/jupyter_client
• https://github.com/requests/requests-oauthlib (this has a requirements.txt but is not used)

Just apps use the requirements.txt like django.

libraries use the setup.py instead requirements.txt and the required libraries are harcoded inside setup.py:

install_requires=["oauthlib>=3.0.0", "requests>=2.0.0"],
extras_require={"rsa": ["oauthlib[signedtoken]>=3.0.0"]},

---

requirements.txt as pom.xml

In order to standardize, you could use the requirements.txt in any of your libraries or apps, modifying the setup.py to read values from requirements.txt instead of hardcoded install_requires

install_requires=["oauthlib>=3.0.0", "requests>=2.0.0"],

With this, you will find a requirements.txt in any python source code and you will be a little close to our Maven in Java

Answer 2

Hopefully this should be helpful. Talks about a pattern similar to one seen in java. Usually in python we would have separate packages that use each other and say they are dependent on each other (or have one pseudo-master package with an init that ties it together).

For the java pattern, an easy way is to have multiple packages under one source control repo. The article explains it in detail.