Trying to install packages with Python 3.7.2 pip causes TSL/SSL errors

Question

I'm running on:

Amazon Linux AMI 2018.03.0

Linux ip-xxx-yy-z-ww 4.14.77-70.59.amzn1.x86_64 #1 SMP Mon Nov 12 22:02:45 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Multi-python environment. It has both Python 2.7.15 and 3.7.2 installed.

Regular (Python2) pip works without a hitch.

When I try to run pip3 install flask I get the following errors:

pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting flask
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/flask/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/flask/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/flask/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/flask/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/flask/
  Could not fetch URL https://pypi.org/simple/flask/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/flask/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
  Could not find a version that satisfies the requirement flask (from versions: )
No matching distribution found for flask
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

The same error appears if I try to install any other package.

pip3 version is 18.1

Adding --trusted-host pypi.org does not help.

Installing ca-certificates did not help

Python 3 installed according to the following guide: https://stackoverflow.com/a/8112006/8826349

EDIT: I now noticed, that when running make on Python 3, it finished successfully, but had the following message:

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_hashlib              _ssl                  _tkinter
_uuid
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

EDIT2:

My issue seems to be identical to this. However the thread does not provide any solutions beyond describing the problem.

I've tried to compile a newer version of openssl using this guide. But newer openssl version did not resolve my issue.

EDIT3:

I was unable to find a solution for this issue.

It seems that Amazon's Linux version does not work well with Python 3.

They do have a different OS, called Amazon Linux 2, where installiing Python 3 is as easy as running yum install python3

However, Amazon Linux 2 has it's own issues. It's Python 2 installation lacks pip. You can install it manually by running:

curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user

But this actually overrides pip3, so then Python 3 remains without a pip.

Answer 1

So, SSLError("Can't connect to HTTPS URL because the SSL module is not available.") in this context indicates that when you compiled python3.7 from sources, it wasn't linked against your system OpenSSL library. The pedantic solution would be to ensure you have the required headers besides the lib, with something like:
$ yum install openssl-devel.x86_64 openssl.x86_64
and recompile from scratch, linking against the OpenSSL lib.

However, as you have rightly noted, installing python3 with yum (from Amazon's own repo, packages tested for Amazon Linux) is way easier and more robust approach in EC2, and will resolve all dependencies. For example, something like this is going to work in Amazon Linux (version 1):

$ yum install python36.x86_64
$ yum install python36-pip.noarch

Usually, Python3.6 is already pre-installed in EC2, but not python36-pip...

This is correct and default behavior that Python3's pip script is called by pip3 command, while pip is for Python2 (if installed - the yum package for python2 is python27-pip.noarch), for example:

$ pip3 -V  
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)  
$ pip2 -V  
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)  
$ pip -V  
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)  

There are also versioned Python commands available in combination with the -m switch (for module) to run corresponding version of pip, for example:
$ python3 -m pip install --user flask

The one you installed via get-pip.py --user has likely landed somewhere in your $HOME/.local/bin directory. The --user switch installs packages just for the current user (in ~/.local/lib/) rather than for the whole system, which is actually a good practice to avoid interfering with the system python packages.

Even better, you could use pip inside a virtualenv or venv isolated environments: there is the python36-virtualenv (installable via yum), and the default 'venv' module built-in Python3 and readily available (python3 -m venv yournewenvname). Once you create and then activate your isolated python environment (a directory), you can just do a simple pip install flask there regardless of python version. Good luck!