differences between users even after using Pipfile and Pipfile.lock with explicit versions

Question

Sorry for the length, this is a pretty intricate pipenv situation.

At my company we are using pipenv (with both Pipfile and Pipfile.lock) to control packages used on different engineers' laptops. This is even more important for us than for most teams because we're also using Zappa to deploy AWS Lambda code, and it apparently packages the dependencies directly from the deployer's laptop to deploy them. So if people's laptops aren't totally aligned in terms of dependencies, we can get different behavior in the cloud depending on who deployed it.

We have found that even after attempting to fully control dependencies with Pipfile and Pipfile.lock, we end up getting different Python packages on our different laptops, as shown by pip freeze and as indicated by errors in deployed code.

Here is the exact process that is showing differences between my laptop and my boss's (the Pipfile code I quote is on multiple lines but I'm condensing it to one line because I'm having trouble with SO formatting):

1. At the very beginning, all we had was a Pipfile with packages specified with wildcards like [requires] python_version = "3.6" [packages] flask = "*". Also, we didn't have a Pipfile.lock, my boss (who was the first coder on this project) had always run --skip-lock
2. To control things better, I started by upgrading our Pipfile to replace the wildcards with explicit versions and also make our Python version more specific, like [requires] python_version = "3.6.4" [packages] Flask = "==1.0.2". To do this, I got a copy of my boss's pip freeze output and copied the versions into the Pipfile where there was a name match with what was listed there (I skipped anything that didn't match because I assumed it was an upstream dependency and we weren't touching that yet). I committed this.
3. We were still having problems, so we decided to start using Pipfile.lock to control upstream dependencies. So my boss created one by running pip install without --skip-lock for the first time, and committed that.
4. I pulled the Pipfile.lock, deleted my environment with pipenv --rm and recreated it with pipenv install
5. We both ran pip freeze and compared outputs, but we both still have a number of differences.

I suppose I can have my boss delete his pipenv environment and reinstall based on the committed Pipfile and Pipfile.lock, but since they are based on his pip freeze I would be a little surprised if that changed anything.

So I'm just wondering: is this behavior truly unexpected? I always thought the combination of pipenv, Pipfile, and Pipfile.lock would guarantee two people have the same packages, as long as every version is locked with ==[version]. Is there anything else we would need to do to get a very exact match?

If it's truly unexpected, the only other thing I can think is that maybe he hadn't run pipenv shell before his pip freeze, but I think he did because things lined up well against the Pipfiles.

Side note: I haven't converted our [dev-packages] in Pipfile to have versions because I'm not sure what that does and I'm assuming it's irrelevant. So those are still like pylint = "*"

ADDITIONAL INFO

Below is some additional info to respond to the comments... but first a couple of interesting things I noticed:

• None of the differences in the first screenshot (for pip freeze diffs) are in the Pipfile.
• It looks like my pip freeze output matches the Pipfile.lock contents, but my boss's doesn't. I think this might explain the differences, but it's a bit surprising that his pip freeze output wouldn't match the Pipfile.lock created by his own pipenv lock, unless the problem is that he ran pipenv lock from outside of pipenv shell.

To respond to the comments... Here is the first part of the diff between the pip freeze outputs (both from within pipenv shell) on my and my boss's laptops:

Here are some diffs in the Pipfile.lock between my and my boss's laptops. The Pipfile.lock was obtained by having him run pipenv lock (outside of pipenv shell although I assume that doesn't matter) and then committing that just now. I then pulled that, deleted my environment with pipenv --rm, ran pipenv install, and got the following differences with the Pipfile.lock that he had just committed. His version is on the left again.

These are all of the differences - one thing I don't get is why we have fewer differences here than with pip freeze. Our Pipfile is still the same between the two of us.

Answer 1

The only way to ensure you share the exact same environment is to synchronize with the same Pipfile.lock, with pipenv sync (optionally pipenv sync --dev).

Pipfile is a helper for humans, an intermediate in the Pipfile.lock creation, it does not ensure that dependencies are exactly the same.

pipenv install calls under the hood 2 pipenv function: lock and sync. pipenv lock will generate a Pipfile.lock from your Pipfile. Even with pinned version in Pipfile, it is possible to have different Pipfile.lock if they are generated at different moments because dependencies of the pinned packages may not be pinned (depending of the publisher). pipenv sync then install the exact packages found in the Pipfile.lock.

To directly install your environment from the dependencies in Pipfile.lock, you have to use pipenv --python 3.6 install --ignore-pipfile, otherwise Pipfile.lock will be regenerated from the Pipfile.

To easily solve your problem, fix a Pipfile.lock version (you can commit it if you use version control, but you do, of course ;), then both use pipenv sync.

Then keep the Pipfile.lock exactly the same as long as you work on minor version, bug fixes... and feel free to regenerate it to get up-to-date dependencies for major versions. In my project, almost all dependencies in the Pipfile are not pinned, and when we start a new major version we update the Pipfile.lock to try fresh dependency versions, test everything, sometimes pin a dependency to a previous version if the latest introduced backward incompatible changes, and we fix the Pipfile.lock until the next major version.