fatal error: ffi.h: No such file or directory on pip2 install pyOpenSSL

Question

One of the steps I always use to install / build my Ansible agent on target nodes was pip2 install pyOpenSSL. This is a requirement for your Ansible client in order to use the standard Ansible module for creating self signed certificates for example openssl_certificate

Since today this no longer works.

I can reproduce this message as follows:

vagrant init generic/ubuntu1804
vagrant ssh
sudo su -
apt-get update && apt-get -y install python python-pip
pip2 install pyOpenSSL

The last step fails with message below. What is correct way to install pyOpenSSL on Ubuntu 18.04 at the moment? Is there a workaround for this problem?

Or beter, is there a Python distribution for Ansible? That includes all dependencies so I don't have to compile build my own Ansible client?

root@ubuntu1804:~# pip2 install pyOpenSSL
Collecting pyOpenSSL
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc72409d5d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc72409d750>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
  Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)
    100% |################################| 61kB 1.0MB/s 
Collecting cryptography>=2.3 (from pyOpenSSL)
  Downloading https://files.pythonhosted.org/packages/e6/68/50698ce24c61db7d44d93a5043c621a0ca7839d4ef9dff913e6ab465fc92/cryptography-2.7-cp27-cp27mu-manylinux1_x86_64.whl (2.3MB)
    100% |################################| 2.3MB 427kB/s 
Requirement already satisfied: six>=1.5.2 in /usr/lib/python2.7/dist-packages (from pyOpenSSL)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/lib/python2.7/dist-packages (from cryptography>=2.3->pyOpenSSL)
Requirement already satisfied: enum34; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography>=2.3->pyOpenSSL)
Requirement already satisfied: ipaddress; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography>=2.3->pyOpenSSL)
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL)
  Downloading https://files.pythonhosted.org/packages/d6/cf/ba7e2df852a2fc807d48b3f7bea46f741830be4f047a0712e6de3e95fb6a/cffi-1.13.0.tar.gz (459kB)
    100% |################################| 460kB 299kB/s 
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL)
  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
    100% |################################| 163kB 280kB/s 
Building wheels for collected packages: cffi, pycparser
  Running setup.py bdist_wheel for cffi ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-YCu4xd/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp_m8aI7pip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/cffi
  copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
  copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
  running build_ext
  building '_cffi_backend' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/c
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-PPrPZj/python2.7-2.7.15=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
  c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
   #include <ffi.h>
            ^~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for cffi
  Running setup.py clean for cffi
  Running setup.py bdist_wheel for pycparser ... done
  Stored in directory: /root/.cache/pip/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511
Successfully built pycparser
Failed to build cffi
Installing collected packages: pycparser, cffi, cryptography, pyOpenSSL
  Running setup.py install for cffi ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-YCu4xd/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-FxUNAq-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/c
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-PPrPZj/python2.7-2.7.15=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
    c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
     #include <ffi.h>
              ^~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-YCu4xd/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-FxUNAq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-YCu4xd/cffi/
root@ubuntu1804:~# 

Answer 1

Install libffi-dev to fix this issue.

apt install libffi-dev

Answer 2

This page is gonna save your life, for all the forthcoming lib issues that you're yet to face,

For Alpine(>=3.6),

use

apk --update --upgrade add gcc musl-dev jpeg-dev zlib-dev libffi-dev cairo-dev pango-dev gdk-pixbuf-dev

Answer 3

cffi requirement set as cffi!=1.11.3,>=1.8 and new version 1.3 has some issue.

You can install 1.12.3 version before pyOpenSSL:

pip2 install cffi==1.12.3
pip2 install pyOpenSSL

PS here you can find issue: https://bitbucket.org/cffi/cffi/issues/424/please-rollback-to-1123