Pip upgrade cannot install packages

Question

I have jumped around for some time now to solve this, and I cannot seem to get it working. I have a docker container where I set up an nvidia image for machine learning. I install all python dependencies. I then start with the pip package installations. I get the first error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

Simple enough I have a certificate to deal with Cisco umbrella. I can then install all packages nice and easy. However to be able to install newest packages I need to upgrade pip, and upgrading works fine. After pip is upgraded to 20.2.3 I suddenly get an error again:

Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)) - skipping

I have then googled around and tried the suggestions I stumbled upon:

Timing

I found that the system time was wrong - it worked for the initial pip version, which was weird. However changing the time did not help the issue.

conf

I added a pip.conf file with global tags for trusted hosts and for certifications. Still the same error persists.

pip install

I have tried with different trusted host flags and also the cert flag, which should already be specified from the conf file - if I understand it correctly. Nevertheless, neither method worked.

What to do

I am kind of at a loss right now, installing the certificate in the container allows me to install packages with pip 9.0.1 (default in the system) after upgrading to pip 20.2.3. I cannot get it to work with any package. I have tried multiple pip versions - but as soon as I upgrade I lose the certificate trying to reinstall it with

ADD Cisco_Umbrella_Root_CA.cer /usr/local/share/ca-certificates/Cisco_Umbrella_Root_CA.crt
RUN chmod 644 /usr/local/share/ca-certificates/Cisco_Umbrella_Root_CA.crt
RUN update-ca-certificates --fresh

Anybody has an idea how this can happen?

UPDATE

Curl

 RUN curl -v -k -H"Host; files.pythonhosted.org" https://files.pythonhosted.org/packages/8a/fd/bbbc569f98f47813c50a116b539d97b3b17a86ac7a309f83b2022d26caf2/Pillow-6.2.2-cp36-cp36m-manylinux1_x86_64.whl
  ---> Running in ac095828b9ec
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::ffff:146.112.56.166...
 * TCP_NODELAY set
 * Connected to files.pythonhosted.org (::ffff:146.112.56.166) port 443 (#0)
 * ALPN, offering h2
 * ALPN, offering http/1.1
 * successfully set certificate verify locations:
 *   CAfile: /etc/ssl/certs/ca-certificates.crt
   CApath: /etc/ssl/certs
 } [5 bytes data]
 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
 } [512 bytes data]
 * TLSv1.3 (IN), TLS handshake, Server hello (2):
 { [85 bytes data]
 * TLSv1.2 (IN), TLS handshake, Certificate (11):
 { [3177 bytes data]
 * TLSv1.2 (IN), TLS handshake, Server finished (14):
 { [4 bytes data]
 * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
 } [262 bytes data]
 * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
 } [1 bytes data]
 * TLSv1.2 (OUT), TLS handshake, Finished (20):
 } [16 bytes data]
 * TLSv1.2 (IN), TLS handshake, Finished (20):
 { [16 bytes data]
 * SSL connection using TLSv1.2 / AES256-GCM-SHA384
 * ALPN, server did not agree to a protocol

From the last line it can be seen that they do not agree on protocol and the communication fails

Answer 1

The steps suggested in the answer and in my question are definitely what one should try. If someone cannot make it working, like me, then in this specific instance it was the IT organisation who had set the information to be proxied to umbrella, and it didn't supprt the ssl scanning/decryption.