Compiling Python modules on Windows x64

Question

I'm starting out some projects in words processing and I needed NumPy and NLTK. That was the first time I got to know easy_install and how to compile new module of python into the system.

I have Python 2.7 x64 plus VS 11 and VS 12. Also Cygwin (the latest one I guess). I could see in the file that compiles using VS that it looks for VS env with the same version as the one that compiled the python code, why? When I hardcoded 11.0 which is my version, numpy failed to build on several strange errors regarding vcvarsall (it found vcvarsall, probably misused it).

Can't I build python binaries on Windows? If not, can I cross compile on Linux for Windows? (using the same method as Google for the Android SDK)

Answer 1

Update: This answer is now over 5 years old! Python-2.7 is about to be deprecated, so what you really need is a Microsoft Visual C compiler for Python-3. Take a look at this Python wiki on MS Windows Compilers. MS Build Tools 2015 with VC-14.0 is what you need to build extensions for Python-3.5 and 3.6. You can either install the older MS build tools 2015 which includes VC-14.0 or the newer MS build tools for 2017 - click the link, then scroll down until you find Build Tools for Visual Studio 2017 - which also includes VC-14.0.

Also if your versions of pip and setuptools are up to date, then you can ignore all of that silly old school MSSDK nonsense. Especially if you are using the VC for Python 2.7 or MS build tools 2015. Since setuptools-24, it just knows where to look for these compilers, hopefully.

Update: As Zooba mentions below, free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft.

Update: Patch vcvarsall.bat to use x64 compilers from SDK v7.0 directly with pip in any shell instead of using SDK shell and setting DISTUTILS_USE_SDK and MSSdk environmental variables as in directions below. See Fix vcvarsall.bat to install Python-2.7 x64 extensions with v90 instead of sdk7.

tl;dr: Use Windows SDK v7.0 compilers, open the SDK shell and call

C:\> setenv /release /x64
C:\> set DISTUTILS_USE_SDK=1
C:\> MSSdk=1

to build Python 2.7.x extensions using distutils, pip or easy_install. See Python x64 Package Extensions with pip, MSVC 2008 Express & SDK 7.

Note: You can install Numpy optimized using Intel MKL from Christoph Gohlke. For virtualenv's, try converting the binary distribution windows installer to a wheel file which can be installed with pip. Building NumPy from source is not trivial. It has become a possibility with the recent introduction and availability of OpenBLAS, a fork of GotoBLAS. There are optimized binaries available for Windows x86 and x86-64 as well as source which is relatively simpler to compile than GotoBLAS or ATLAS.

Note: In the past I hadn't recommended compiling NumPy from source because you had to optimize BLAS for your platform which was very time-intensive, and the reference BLAS (FORTRAN) or CBLAS (C/C++) implementations were relatively low-performing. Also NumPy depends on LAPACK which also depends on BLAS an additional hurdle to building NumPy. However OpenBLAS binaries are already compiled with LAPACK, so this obstacle has already been removed. See Carl Kleffner's static MinGW-w64 toolchains to build NumPy with OpenBLAS on Windows - he has resolved some of the ABI incompatibilities and issues linking to the correct Microsoft Visual C Common Runtime library (msvcr90.dll for Python-2.7).

So onward to the main question about installing Python extensions on Windows 7 x64.

The recommended way to compile extensions is to use the same compiler used to compile the Python shared library [1-3]. The official Python 2.7.x for Windows was compiled using Microsoft Visual C++ Compilers version 9.0 [4] (aka MSVC90 or simply VC90) from Microsoft Visual Studio 2008 Professional, which you might be able to get for free from Microsoft DreamSpark. Evidentally the x64 compilers are not installed by default, so make sure they are installed along with the x86 compilers. Note: MS Visual Studio 2008 may no longer be available, but MS Visual Studio 2010 will let you use the MSVC90 toolchain if installed, which can be installed from Windows SDK7.

You can tell what version your Python was built with by looking at the header when you execute the python interpreter from a command line. e.g.: Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32 means that it was built with VS 2008 C++. You can build Python yourself from source, but this is also an arduous task. In general it is not recommended to mix compiler runtimes, although in practice you may find that they still work.

If you have your heart set on using GNU GCC, then you would have to use mingw-w64 because MinGW is only for native x86 applications, not AMD64/x86-64. You will need to

• create a distutils config file to specify the mingw compiler,
• remove the -mno-cygwin bug from distutils.cygwinccompiler.Mingw32ccompiler and
• export a library definitions file using either pexports.exe from msys or gendef.exe from msys2 and make a static library libpython27.a using dlltool.exe (msys/msys2) (but the newest releases of Official Python for Windows already have this file in the Python27\lib folder, thanks!).

However in the end you will still need to link against the runtime that python was built with, msvcr90.dll, so you will need Visual C++ 2008 redistributable. Note: Windows GCC from mingw-w64 uses msvcrt.dll which is not the same as msvcr90.dll or later.

If you want to do it for free you can use Microsoft Visual C++ 2008 Express SP1 but you will need to add the Windows SDK 7 for .NET Frameworks 3.5 SP1 because the express version does not have the x64 compilers. The procedure for installing x64 C++ extensions with VS2008 Express & SDK 7 are very similar to the those on the cython site for windows x64 extensions. Note: MS Visual Studio 2008 Express is no longer available or supported.

FYI: You do not necessarily need Visual Studio to build using Microsoft compilers. They are available free with the appropriate SDK package. CL.EXE is the compiler executable, but you will have to set whatever platform options that are normally configured by Autotools or some other tool such as CMAKE. CMAKE plays well with MSVC, but Autotools AFAIK doesn't work with MSVC and would require some POSIX environment and utilities which on Windows are available in MSYS.

For many Python packages that use distutils or setuptools, they can compile extensions using Windows SDK 7 by following the directions that are posted in various places through the reference documentation and wikis:

1. From the Start Menu select All Programs then Microsoft Windows SDK v7.0 and start CMD Shell to open a command window optimized for Windows SDK.

2. Step #1 is the equivalent of typing the following in the Run box from the Start Menu or from a Command Prompt (aka C:\Windows\System32\cmd.exe):

   %COMSPEC% /E:ON /V:ON /K "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.0\Bin\SetEnv.cmd"
   

   NOTE: The flags /E:ON, which enables command extensions, and /V:ON, which enables delayed variable expansion, are both necessary to for SetEnv.cmd to function, or you will get an message that the x64 compilers are not installed, &c.

3. Then type setenv /Release /x64 which will set the SDK environment variables specifically for Windows 7 x64 release (vs debug or x86 which are the default).

4. Type set DISTUTILS_USE_SDK=1 hit return and then type set MSSdk=1 and return to tell distutils.msvccompiler that you are using the SDK, and that the SDK will determine all of the environment variables.
5. Now use your installation method of choice:
   • pip install pyyaml ntlk which is the recommended way, see ntlk, but you must have setuptools and pip installed.
   • python setup.py install for each downloaded, extracted tarball
   • easy_install pyyaml ntlk which is the old way and the only way to install eggs.

[1] Building C and C++ Extensions on Windows
[2] distutils.msvccompiler — Microsoft Compiler
[3] Python Dev Guide: Getting Started: Windows
[4] What version of visual studio is this python compiled with