Alternatives to keeping large lists in memory (python)

Question

If I have a list(or array, dictionary....) in python that could exceed the available memory address space, (32 bit python) what are the options and there relative speeds? (other than not making a list that large) The list could exceed the memory but I have no way of knowing before hand. Once it starts exceeding 75% I would like to no longer keep the list in memory (or the new items anyway), is there a way to convert to a file based approach mid-stream?

What are the best (speed in and out) file storage options?

Just need to store a simple list of numbers. no need to random Nth element access, just append/pop type operations.

Answer 1

You can try blist: https://pypi.python.org/pypi/blist/

The blist is a drop-in replacement for the Python list the provides better performance when modifying large lists.

Answer 2

Did you check shelve python module which is based on pickle?

http://docs.python.org/library/shelve.html