Why Python `Memory Error` with list `append()` lots of RAM left

Question

I am building a large data dictionary from a set of text files. As I read in the lines and process them, I append(dataline) to a list.

At some point the append() generates a Memory Error exception. However, watching the program run in the Windows Task Manager, at the point of the crash I see 4.3 GB available and 1.1 GB free.

Thus, I do not understand the reason for the exception.

Python version is 2.6.6. I guess, the only reason is that it is not able to use more of the available RAM. If this is so, is it possible to increase the allocation?

Answer 1

If you're using a 32-bit build of Python, you might want to try a 64-bit version.

It is possible for a process to address at most 4GB of RAM using 32-bit addresses, but typically (depending on the OS), one gets much less. It sounds like your Python process may be hitting this limit. 64-bit addressing removes this limitation.

edit Since you're asking about Windows, the following page is of relevance: Memory Limits for Windows Releases. As you can see, the limit per 32-bit process is 2, 3 or 4GB depending on the OS version and configuration.

Answer 2

If you're open to restructuring the code instead of throwing more memory at it, you might be able to get by with this:

data = (processraw(raw) for raw in lines) 

where lines is either a list of lines or file.xreadlines() or similar.