Cannot allocate 1.6 GB in Python

Question

This code produces a MemoryError :

from pylab import complex128
import numpy

x = numpy.empty(100000000, dtype=complex128)    # 100 millions complex128

I have Win7 64 with 8 GB RAM (at least 5.3 GB free when running this code). I'm using Python 2.7 (Anaconda) and I think it is the 32 bits version. Even with 32bits, we should be able to handle 1.6 GB !

Do you know how to solve this ?

PS : I expected an array of 100 millions items, each of one using 16 bytes (128 bits) to use 16 * 100 millions = 1.6 GB. This is confirmed by :

x = numpy.empty(1000000, dtype=complex128)    # 1 million here
print x.nbytes
>>> 16000000        # 16 MB

Answer 1

The problem was solved with Python 64bit.

It's even possible to create a single array of more than 5 GB.

Note : when I create an array which should use 1 600 000 000 bytes (with 100 million items in a complex128 array), the actual memory usage is not "much" more : 1 607 068 KB...

Answer 2

I know it's old question. And I know there are many similar questions. ex. Memory for python.exe on Windows 7 python 32 - Numpy uses half of the available memory only? But none of them seems to really solve the issue.

Using the hint given here https://stackoverflow.com/a/18282931/566035, I think I finally fixed this issue.

First, you need to install "Microsoft Visual C++ Express Edition 2008". You can follow the instructions given here: http://blog.victorjabur.com/2011/06/05/compiling-python-2-7-modules-on-windows-32-and-64-using-msvc-2008-express/

The download URL for Microsoft Visual C++ Express Edition 2008 in the above blog article is dead. But, you can find the URL here Visual C++ 2008 Express Download Link Dead?.

(EDIT) I confirmed that the linker that comes with msvc-2010-express also works.

Then, launch Visual Studio 2008 Command Prompt from start menu -> Microsoft Visual C++ 2008 Express Edition -> Visual Studio Tools - > Visual Studio 2008 Command Prompt

Then do these commands:

cd bin
editbin.exe /LARGEADDRESSAWARE "C:\Python27\python.exe"

This will set IMAGE_FILE_LARGE_ADDRESS_AWARE flag in the python executable. With this magic, 32 bit python will use up to 4 GB (instead of ~2 GB on windows).

According to MSDN:

On 64-bit editions of Windows, 32-bit applications marked with the IMAGE_FILE_LARGE_ADDRESS_AWARE flag have 4 GB of address space available.

Now,

x = numpy.empty(100000000, dtype=complex128)

actually works on my Windows 7 64-bit PC with 32 bit Python 2.7.

I really hope the official python binary to be shipped with this FLAG already set, as there is no harm in doing so but huge benefit!

As MSDN says:

Setting this flag and then running the application on a system that does not have 4GT support should not affect the application.