Memory Efficiency of NumPy

Question

While learning NumPy, I came across its advantage that,

NumPy requires less memory than traditional list.

import numpy as np
import sys

# Less Memory

l = range(1000)
print(sys.getsizeof(l[3])*len(l))

p = np.arange(1000)
print(p.itemsize*p.size)

this looks convincing, but than when I try,

print(sys.getsizeof(p[3])*len(p))

It shows higher memory size than list.

Can someone help me out understanding this behavior.

Answer 1

First off all, as mentioned in comments getsizeof() is not a good function to relay on for this purpose, because it does not have to hold true for third-party extensions as it is implementation specific. Also, as mentioned in documentation, if you want to find the size of containers and all their contents, there is a recipe available at: https://code.activestate.com/recipes/577504/.

Now, regarding the Numpy arrays, it's very important to know how Numpy determines its arrays' types. For that purpose, you can read: How does numpy determin the array's dtype and what it means?

To sum up, the most important reason that Numpy performs better in memory managements is that it provides a wide variety of types that you can use for different kinds of data. You can read about Numpy's datatypes here: https://docs.scipy.org/doc/numpy-1.14.0/user/basics.types.html. Another reason is that Numpy is a library designed to work with matrices and arrays and for that reason there are many under the hood optimizations on how their items consume the memory.

Also, it's note worthy that Python provides an array module designed to perform efficiently by using constrained item types.

Arrays are sequence types and behave very much like lists, except that the type of objects stored in them is constrained. The type is specified at object creation time by using a type code, which is a single character.