python-3.7 build from source, pip not being installed in altroot/DESTDIR

Question

I have a custom linux distribution, and am building python 3.7.2 from source. The build is configured to install into an alternate root directory, from which it will be packaged for installation.

This process works fine when run on a system that does not yet have python3 installed. But, when I run the same build process on a host that has a previously built copy of the same package on it, ensurepip fails to install pip or setuptools into the alternate installation directory, because it finds them on the system itself.

Software is extracted and configure called as follows:

mkdir -p /data/buildspace/work/python3-x86_64
/bin/tar xf /data/sources/python3/Python-3.7.2.tgz \
     -C /data/buildspace/work/python3-x86_64
( cd /data/buildspace/work/python3-x86_64/Python-3.7.2 ; \
     [ -f ./configure ] && CC="gcc" CXX="g++" ./configure \
     --prefix=/usr --libdir=/usr/lib
     --sysconfdir=/etc --bindir=/usr/bin  || echo "No './configure' script")

Then make install is called with DESTDIR specified:

make -C /data/buildspace/work/python3-x86_64/Python-3.7.2
    install DESTDIR=/data/buildspace/work/python3-x86_64/root 

When building initially, before the build system has python3 installed, I see in the build output/log:

if test "xupgrade" != "xno"  ; then \
    case upgrade in \
            upgrade) ensurepip="--upgrade" ;; \
            install|*) ensurepip="" ;; \
    esac; \
     ./python -E -m ensurepip \
            $ensurepip --root=/data/buildspace/work/python3-x86_64/root/ ; \
fi
Looking in links: /tmp/tmpa9_lumsc
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-18.1 setuptools-40.6.2

However, when building on a system that has python3 installed, I can't find any way to prevent it from considering the pip in /usr/lib/python3.7/site-packages. Running the same configure will yield:

Looking in links: /tmp/tmp086yebrw
Requirement already up-to-date: setuptools in /usr/lib/python3.7/site-packages (40.6.2)
Requirement already up-to-date: pip in /usr/lib/python3.7/site-packages (18.1)

Then, I tried --with-ensurepip=install, but that only changed the output slightly:

Looking in links: /tmp/tmpl0x88vr9
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (40.6.2)
Requirement already satisfied: pip in /usr/lib/python3.7/site-packages (18.1)

I don't know if this is a bug in ensurepip, where it should not consider the default python library directory when run with --root specified, or if there is something I'm missing for this "install into a whole new space".

I looked at the idea of using venv, but I want the resulting binaries to be able to run in normal locations on the system after package and install, so I think that may be the wrong direction.

Appreciate any pointers. Thanks.

Answer 1

It's three years later, no answer was provided, and this bug has existed within ensurepip since 2015, so for posterity's sake, I figured I'd share the solution that worked for me:

Use get-pip instead, and pass the --ignore-installed flag when calling get-pip.py to prevent overwriting a preexisting system installed pip.

$ cd <python_bin_dir>
$ curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ python get-pip.py --ignore-installed