Best practice for using pipenv Pipfile and Pipfile.lock across multiple platforms

Question

Is there a best practice for using pipenv for deterministic builds when you're going to be developing and running application code on multiple platforms (i.e. Windows, Linux, and Mac)?

For instance, the requirements for pytest defines the atomicwrites library as a conditional dependency if you are installing pytest in a Windows-based Python environment.

However, if I define...

[dev-packages]
pytest = "*"

as a requirement in my project's Pipfile and run pipenv install --dev to generate my initial Pipfile.lock in a Linux-based Python environment, then atomicwrites is neither installed nor specified in any way in the resulting Pipfile.lock file.

Later, after I git commit my new Linux-generated Pipfile.lock, either I or someone else will eventually pull that committed Pipfile.lock file down onto their Windows machine and will run pipenv install --dev to generate their own local pipenv environment.

• However, when they go to run the pytest test-runner, it will fail because atomicwrites will not have been installed in their pipenv environment, therefore the pytest command will fail because of the missing dependency.

• What's more, my Windows test-build will also fail when using a CI service like GitHub Actions or Azure Pipelines because pipenv will fail to install the atomicwrites dependency there too (because it will not be specified in the repo's Pipfile.lock specifications).

This pytest example is a super simple example of this issue. In this case, it'd be easy enough just to add atomicwrites as one of my [dev-packages] requirements in my Pipfile so that it gets installed regardless of the platform, or to even add sys_platform = "== 'win32'" to specify that it should only be installed by pipenv on Windows platforms.

However, these platform-conditional dependencies become a much harder issue to deal with when my project has many dependencies, all with their own platform-conditional dependencies.

I've seen this issue discussed in a couple different locations, such as here, and here.

However, I have yet to find any straightforward method for dealing with this (short of not using pipenv or deleting the Pipfile.lock file prior to running pipenv install --dev on a different platform).

Do any pipenv users out there have a recommended best practice for dealing with this multiple-os Pipfile.lock install issue?

Answer 1

This is obviously too late to be helpful to you, but I have found that generating the lock file in these multiple environments while keeping outdated versions resolves most of the problems with pipenv.

For example, our CI/CD uses GitHub images, so in a Windows Subsystem for Linux shell:

pipenv install --dev

Then, in a Windows shell

pipenv install --dev --keep-outdated

However, running in Windows can sometimes pin a dependency to that platform (colorama currently does this as of this answer's writing). To avoid that, you can regenerate the lock file again in WSL:

pipenv lock --dev --keep-outdated

This will keep the "outdated" packages from a Windows-only environment, but will generally fix the platform conditionals.

Note that the dance above is ultimately not foolproof--I found this question because this exact method was not working for atomicwrites. However, it seems to resolve the vast majority of issues and those that cannot generally can be solved by adding the package to your dependencies manually.