Are poetry.lock files OS independent?

Question

I am creating a poetry.lock file on my Mac. Then, I am using it to build a Docker image based on Debian. My question is the following: is there any guarantee that the exact packages will be found by the Debian image? I might be mistaken but I remember packages might not exist in every version for every OS. That’s seem reasonable when thinking about os specific packages (windows vs Unix). But how about Debian and Mac (both Unix based)? Should I expect at the least the majority of the packages to exist in the same version?

Answer 1

Broadly, they are portable to any OS, yes.

But you must understand this has "nothing" to do with poetry nor pipenv nor pip but with the fact that each package version can be distributed for multiple platforms.

It is pip who will pick a particular wheel that matches the compatible platform tags for the system issuing the install (see the output of pip debug).

• For most pure python packages: only a wheel and a source distribution (tarball) can be found in the indexes, and pip will elect the wheel unless you tell it otherwise. So, the install will be pretty much the same across different OS. Only the python interpreter will be different.

• For packages featuring C (or whatever) extensions: the package will have been precompiled for a series of platforms and pip will pick the best mach for your system and python version. If no prebuilt wheel is compatible with your system, pip is forced to fallback to the source distribution and compile it, at which point you will be required whatever building toolchain that package needs (a C compiler, CPython headers and whatnot).

The main purpose of a lockfile is a reproductible install

This is achieved through

• Version pinning: if no platform tag was compatible and the source distribution failed to build, pip would try to pick the next greatest version compatible with the requirement version range. In presence of a lockfile, that range is exactly one only version. Thus the installation would fail with no chances of picking another version.
• Artifact hashing: the downloaded artifacts are hashed and checked against the saved hashes in the lockfile. This simply prevents a rogue package index from tampering the source code nor the metadata of formerly published wheels or tarballs.

Code from binary extensions will be equivalent yet inherently different across platforms

It is reasonable to trust the binary extensions will be equivalent and will come from the very same source code, executing the same behaviour with only potential platform-specific bugs. Yet, strictly speaking the compiled code will be usually different.

If you end up compiling the packages:

• for different platforms, the compilation will definetely be different
• for a given platform but different hardware, the compilation will most usually still be different
• for similar systems, the compilation toolchain can be different or be configured to perform different optimizations
• for the same system, the compilation might not be reproductible

If you end up downloading a prebuilt wheel:

• you only trust the package maintainers to have built all public wheels for different platforms out of the same source code version
• prebuilt binary extensions will be different for each platform tag set, and might present different bugs or issues

This is rarely the case for pure Python packages: theoretically possible, but maintainers will rather build a single wheel with compatibility boilerplate than craft different code bases for each Python version, even for Python-2&3 with two actual py{2,3}-none-any public wheels.

Example

watchdog = [
    {file = "watchdog-2.1.6-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:9693f35162dc6208d10b10ddf0458cc09ad70c30ba689d9206e02cd836ce28a3"},
    {file = "watchdog-2.1.6-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:aba5c812f8ee8a3ff3be51887ca2d55fb8e268439ed44110d3846e4229eb0e8b"},
    {file = "watchdog-2.1.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4ae38bf8ba6f39d5b83f78661273216e7db5b00f08be7592062cb1fc8b8ba542"},
    {file = "watchdog-2.1.6-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:ad6f1796e37db2223d2a3f302f586f74c72c630b48a9872c1e7ae8e92e0ab669"},
    {file = "watchdog-2.1.6-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:922a69fa533cb0c793b483becaaa0845f655151e7256ec73630a1b2e9ebcb660"},
    {file = "watchdog-2.1.6-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:b2fcf9402fde2672545b139694284dc3b665fd1be660d73eca6805197ef776a3"},
    {file = "watchdog-2.1.6-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:3386b367e950a11b0568062b70cc026c6f645428a698d33d39e013aaeda4cc04"},
    {file = "watchdog-2.1.6-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:8f1c00aa35f504197561060ca4c21d3cc079ba29cf6dd2fe61024c70160c990b"},
    {file = "watchdog-2.1.6-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:b52b88021b9541a60531142b0a451baca08d28b74a723d0c99b13c8c8d48d604"},
    {file = "watchdog-2.1.6-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:8047da932432aa32c515ec1447ea79ce578d0559362ca3605f8e9568f844e3c6"},
    {file = "watchdog-2.1.6-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e92c2d33858c8f560671b448205a268096e17870dcf60a9bb3ac7bfbafb7f5f9"},
    {file = "watchdog-2.1.6-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:b7d336912853d7b77f9b2c24eeed6a5065d0a0cc0d3b6a5a45ad6d1d05fb8cd8"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_aarch64.whl", hash = "sha256:cca7741c0fcc765568350cb139e92b7f9f3c9a08c4f32591d18ab0a6ac9e71b6"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_armv7l.whl", hash = "sha256:25fb5240b195d17de949588628fdf93032ebf163524ef08933db0ea1f99bd685"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_i686.whl", hash = "sha256:be9be735f827820a06340dff2ddea1fb7234561fa5e6300a62fe7f54d40546a0"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_ppc64.whl", hash = "sha256:d0d19fb2441947b58fbf91336638c2b9f4cc98e05e1045404d7a4cb7cddc7a65"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:3becdb380d8916c873ad512f1701f8a92ce79ec6978ffde92919fd18d41da7fb"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_s390x.whl", hash = "sha256:ae67501c95606072aafa865b6ed47343ac6484472a2f95490ba151f6347acfc2"},
    {file = "watchdog-2.1.6-py3-none-manylinux2014_x86_64.whl", hash = "sha256:e0f30db709c939cabf64a6dc5babb276e6d823fd84464ab916f9b9ba5623ca15"},
    {file = "watchdog-2.1.6-py3-none-win32.whl", hash = "sha256:e02794ac791662a5eafc6ffeaf9bcc149035a0e48eb0a9d40a8feb4622605a3d"},
    {file = "watchdog-2.1.6-py3-none-win_amd64.whl", hash = "sha256:bd9ba4f332cf57b2c1f698be0728c020399ef3040577cde2939f2e045b39c1e5"},
    {file = "watchdog-2.1.6-py3-none-win_ia64.whl", hash = "sha256:a0f1c7edf116a12f7245be06120b1852275f9506a7d90227648b250755a03923"},
    {file = "watchdog-2.1.6.tar.gz", hash = "sha256:a36e75df6c767cbf46f61a91c70b3ba71811dfa0aca4a324d9407a06a8b7a2e7"},
]
weasyprint = [
    {file = "weasyprint-54.1-py3-none-any.whl", hash = "sha256:27c078ded67a43c9a05c349eda01ea327805d48e5c3ca3b704f57eb82bd78592"},
    {file = "weasyprint-54.1.tar.gz", hash = "sha256:fa57db862e06bd01c5e7d82dad399b3b9952a39827023c17bee9b1c061ff1bbd"},
]

In this fragment of a poetry.lock file you can see the hashes for watchdog==2.1.6 and weasyprint==54.1. For weasyprint, the wheel tagged py3-none-any will probably be the wheel to be actually downloadad in any system.

Yet, for watchdog, the elegible wheel will depend both on your OS and your hardware, aside from your python version!

• Any x86_64 linux box will pick watchdog-2.1.6-py3-none-manylinux2014_x86_64.whl
• An arm64 MacOS box with CPython-3.8 will pick watchdog-2.1.6-cp38-cp38-macosx_11_0_arm64.whl
• An arm64 MacOS box with PyPy-3.7 will be forced to build watchdog-2.1.6.tar.gz, which could lead to the whole installation failing if it can't achieve that.
• A future MacOS with CPython-3.11 ill also need to build watchdog-2.1.6.tar.gz

You can also check those hashes in PyPI's website https://pypi.org/project/watchdog/2.1.6/#files