Fatal error when trying to install PyCrypto on OS X El Capitan

Question

I am trying to install PyCrypto on OS X 10.11.3 (El Capitan). I am using Python 3.5.1. I downloaded the gzip file from https://pypi.python.org/pypi/pycrypto and decompressed it. Then I ran python setup.py build like the instructions said and it appeared to do something, then it produced this output:

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -fwrapv -Wall -Wstrict-prototypes -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/MD2.c -o build/temp.macosx-10.6-intel-2.7/src/MD2.o
src/MD2.c:30:10: fatal error: 'string.h' file not found
#include <string.h>
         ^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

I tried python3 setup.py build and got some very similar output:

/usr/bin/clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -fno-common -dynamic -fwrapv -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c src/MD2.c -o build/temp.macosx-10.6-intel-3.5/src/MD2.o
src/MD2.c:30:10: fatal error: 'string.h' file not found
#include <string.h>
         ^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

I tried Googling to figure out what to do, but I couldn't find anything useful. How can I install PyCrypto?

EDIT: I also tried several other things like pip install pycrypto and sudo pip3 install pycrypto and they didn't work. @l'L'l helped me get it to work by doing several strange, complex things that I never would have though have myself. They are summarized in the answer below.

Answer 1

Overview:

The manual build you're trying looks like it might be failing because it's referencing the OS X 10.6 SDK, which you likely don't have, and is outdated for the most part. Also, SDKs are now stored in a completely different location than when the 10.6 SDK was in it's prime.

New SDKs location:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/

Old SDKs location:

/Developer/SDKs/

Non-existent / outdated SDK:

Since it appears that when trying to build PyCrypto it's referencing the MacOSX10.6.sdk there are several things to consider:

1. Why does it reference an outdated SDK
2. Where is the SDK it's referencing set
3. What should be done to correct the issue

Unless we audit the source code carefully we might not know exactly where the incorrect flags are set, but we can do our best to work with the information we have. From the error we can see that there are several instances where the 10.6 SDK's name pops up:

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -fwrapv -Wall -Wstrict-prototypes -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library

Building from source:

/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/MD2.c -o build/temp.macosx-10.6-intel-2.7/src/MD2.o
src/MD2.c:30:10: fatal error: 'string.h' file not found
#include <string.h>
                 ^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

Analyzing this we can see that the PyCrypto's MD2.c file is trying to be built using the flag -isysroot /Developer/SDKs/MacOSX10.6.sdk. It might be worth trying pip instead:

Installing with pip:

...
fatal error: 'string.h' file not found #include <string.h>
...

Same error; we should probably find out if the <string.h> header even exists on the system — Let's make a quick test C application to find out:

Testing the C headers:

$ echo "#include <string.h>
#include <stdio.h>
int main() { printf(\"TEST\n\"); return 0; }" > t.c
$ clang t.c -o t
$ ./t
TEST

It's apparent the header does exist because the test worked fine. This tells us that the problem is more likely related directly to the 10.6 SDK (which doesn't seem to exist on the system).

Symlinking (non-existing) 10.6 SDK to 10.11 SDK:

Since we haven't determined where the SDK is actually getting set we'll go ahead and try to create symlinks so that any reference of the old 10.6 SDK links to the latest SDK (10.11 at this time):

$ cd /Developer/SDKs
$ sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk MacOSX10.6.sdk

We can verify the symlink by issuing the following command:

$ ls -lat
total 8
drwxr-xr-x  3 root  wheel  102 Feb 21 15:54 .
lrwxr-xr-x  1 root  wheel   99 Feb 21 15:54 MacOSX10.6.sdk -> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
drwxr-xr-x  3 root  wheel  102 Feb 21 15:52 ..

Now that we've successfully created the symlink let's try installing PyCrypto with pip once more:

$ sudo pip install pycrypto
Collecting pycrypto
  Downloading pycrypto-2.6.1.tar.gz (446kB)
    100% |████████████████████████████████| 446kB 1.2GB/s 
Installing collected packages: pycrypto
  Running setup.py install for pycrypto ... done
Successfully installed pycrypto-2.6.1

No errors! It looks like our problem is solved! Well, almost...

We still need to figure out what is responsible for setting the wrong (10.6) SDK during builds. Let's use the xcrun tool to see what the defaults are set at:

$ xcrun --show-sdk-version
10.11

The system default SDK is set to 10.11, so it must be get set incorrectly to 10.6 by Python, PyCrypto, or some other anomaly we might not have considered.

UPDATE:

After doing some recon it was discovered that Python 3 appears to be built with the OS X 10.6 SDK. In addition it's also setting the SDK to 10.6 and setting the (outdated) path in numerous places throughout the Python_Framework. There are so many references I won't bother listing them all, although here's an example:

Python_Framework Folder/Versions/3.5/lib/python3.5/config-3.5m/Makefile:79:CONFIGURE_CFLAGS= -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk

I can only assume the developers were trying to be as backward compatible as possible, however, it's unfortunately breaking forward compatibility in the process.

---

Notes:

Installing Python packages with pip can make life much easier in a lot of ways (package management, updating, uninstalling, etc.). For example installing PyCrypto should just be a matter of issuing the command:

$ sudo pip install pycrypto

If you have multiple Python's you can use the version number to install for that Python accordingly:

$ sudo pip3.5 install pycrypto

↳ https://pip.pypa.io/en/stable/installing/