Getting Memory Error when trying to plot large array with matplotlib

Question

I would like to plot an array of 20 millions object, I have 8GB RAM and still I get the following error when I run the following lines:

import matplotlib.pyplot as plt
import numpy as np

d = np.arange(200000000)
plt.plot(d)
plt.show()

Error:

Traceback (most recent call last):
...
  File "C:\Python27\lib\site-packages\matplotlib\axes.py", line 317, in _grab_next_args
    for seg in self._plot_args(remaining, kwargs):
  File "C:\Python27\lib\site-packages\matplotlib\axes.py", line 292, in _plot_args
    x = np.arange(y.shape[0], dtype=float)
MemoryError

Answer 1

Due to physical limitations of displays and plotters, you won't be able to plot 20,000,000 points anyway. So you could reduce your array by sampling it or by using means of slices:

>>> m = 20000000
>>> a = np.arange(m)
>>> n = 100 # <- reducing to 100 points
>>> s = m/n # <- size of slices to compress
>>> reduced = []
>>> for i in xrange(n):
...     slice = a[i*s:(i+1)*s]
...     reduced.append(np.mean(slice))
>>> reduced
[99999.5, 299999.5, ..., 19699999.5, 19899999.5]

.. assuming np.mean makes sense on the objects you're plotting.

Answer 2

MemoryError isn't lying--you had a failed memory allocation. This isn't really that unreasonable considering you're trying to plot 200 million points (note that the number you posted was 200 million, not 20 million).

It seldom or never makes sense to plot millions of points. When I have large datasets, I preprocess my data so that I am plotting no more than thousands of points. A simple regular sample would be fine for this for data like you have, but peakfinding can be necessary for great depictions of other data.