Different python versions when exec and when building image with singularity, the same official python docker image

Question

I'm trying to create a singularity image based on docker://python:3.7-buster.

In the %post section, when I run /usr/bin/env python3 a different version of python is used than when I singularity exec the same thing.

Here is my test definition file:

$ cat test.def 
Bootstrap:docker
From:python:3.7-buster

%post
        /usr/bin/env python3 --version

And here is what happens when I try to build it:

$ sudo singularity build test.sif test.def 
INFO:    Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:4a56a430b2bac33260d6449e162017e2b23076c6411a17b46db67f5b84dde2bd
Skipping fetch of repeat blob sha256:4b5cacb629f5c5323a32103e665756e5d50fe133b3db72d444f370566b713a6a
Skipping fetch of repeat blob sha256:14408c8d4f9a59a5da8f4cc40650be9a8d0991fa1ce1b2fb2767f289a9cc410d
Skipping fetch of repeat blob sha256:ea67eaa7dd42136287337f879ef20b4ee73baaa108d833d267ef99dd787cdcbf
Skipping fetch of repeat blob sha256:4d134ac3fe4b8dd8136d9e7acbb2708ead2154185b27c09ae62ca099070cfb27
Skipping fetch of repeat blob sha256:4c55f6f5d7f035e446f063331d9160bb00ed3da4632105ef5adedee3317c902f
Skipping fetch of repeat blob sha256:6ae475e50652d8ee1a2fdeb59ccce81d14c8c20e0fdfe94f22f1c69bd3e3befb
Skipping fetch of repeat blob sha256:6f41526442299286e270923d6cca3a516c3e1850f7e06c3facc0df7da8a5afbc
Skipping fetch of repeat blob sha256:6933d3d4604265f0c8f2a3806222749809c62b6e6a757d1f85720fa81622496d
Copying config sha256:5a5fb77dac35d62c5b062fc35b3b69e61ae68385fb4278ce6076532c3e50e316
 7.47 KiB / 7.47 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
2019/09/16 11:07:07  info unpack layer: sha256:4a56a430b2bac33260d6449e162017e2b23076c6411a17b46db67f5b84dde2bd
2019/09/16 11:07:09  info unpack layer: sha256:4b5cacb629f5c5323a32103e665756e5d50fe133b3db72d444f370566b713a6a
2019/09/16 11:07:09  info unpack layer: sha256:14408c8d4f9a59a5da8f4cc40650be9a8d0991fa1ce1b2fb2767f289a9cc410d
2019/09/16 11:07:09  info unpack layer: sha256:ea67eaa7dd42136287337f879ef20b4ee73baaa108d833d267ef99dd787cdcbf
2019/09/16 11:07:11  info unpack layer: sha256:4d134ac3fe4b8dd8136d9e7acbb2708ead2154185b27c09ae62ca099070cfb27
2019/09/16 11:07:18  info unpack layer: sha256:4c55f6f5d7f035e446f063331d9160bb00ed3da4632105ef5adedee3317c902f
2019/09/16 11:07:18  info unpack layer: sha256:6ae475e50652d8ee1a2fdeb59ccce81d14c8c20e0fdfe94f22f1c69bd3e3befb
2019/09/16 11:07:19  info unpack layer: sha256:6f41526442299286e270923d6cca3a516c3e1850f7e06c3facc0df7da8a5afbc
2019/09/16 11:07:19  info unpack layer: sha256:6933d3d4604265f0c8f2a3806222749809c62b6e6a757d1f85720fa81622496d
INFO:    Running post scriptlet
+ /usr/bin/env python3 --version
Python 3.7.3
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

The system version is used instead the one provided by the docker image, in contrast with what happens when I just exec the same command:

$ singularity exec docker://python:3.7-buster /usr/bin/env python3 --version
Python 3.7.4

What happens?

I tried to use $(which python3) instead of /usr/bin/env python3, and it is still the same version during the %post section (with singularity exec, it's the host system's version that is used).

My goal is actually to be able to install some personal python packages that I pull from a git repository, and that get installed using install scripts that run /usr/bin/env python3 -m pip install -e .

I noticed there was a problem because the system version of python 3 did not include pip.

Answer 1

You're getting different versions of python because the environments are different.

If you add echo $PATH in %post you'll get: /bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

If you run singularity exec docker://python:3.7-buster bash -c 'echo $PATH' you'll get: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

/usr/local/bin/python3 is the newer version that the docker container compiled for you and the PATH in %post prioritizes /usr/bin over /usr/local/bin. You can adjust the PATH variable at the beginning of the %post block to fix the issue.

As to the why it is using a different PATH there, I'm not sure. Might be worth raising an issue at the github repo.