MatPlotLib, datetimes, and TypeError: ufunc 'isfinite' not supported for the input types…

Question

Here is a tiny piece of code that produces a filled in region between two lines of a graph:

import matplotlib.pyplot as plt
import numpy as np

x = np.arange(0.0, 2, 0.01)
y1 = np.sin(2 * np.pi * x)
y2 = 1.2 * np.sin(4 * np.pi * x)

fig, ax1 = plt.subplots(1, 1, sharex=True)

# Test support for masked arrays.
ax1.fill_between(x, 0, y1)
ax1.set_ylabel('between y1 and 0')
y2 = np.ma.masked_greater(y2, 1.0)
ax1.plot(x, y1, x, y2, color='black')
ax1.fill_between(
    x, y1, y2, where=y2 >= y1,
    facecolor='green',
    interpolate=True)
ax1.fill_between(x, y1, y2, where=y2 <= y1, facecolor='red', interpolate=True)
ax1.set_title('Now regions with y2>1 are masked')

# Show the plot.
plt.show()

It looks like so:

Now, changing the start so that x is now a collections of date times objects like so:

import datetime

x1 = np.arange(0.0, 2, 0.01)
now = np.datetime64(datetime.datetime.now())
x = np.array([now - np.timedelta64(datetime.timedelta(seconds=i)) for i in range(200)])
y1 = np.sin(2 * np.pi * x1)
y2 = 1.2 * np.sin(4 * np.pi * x1)

yields:

Traceback (most recent call last):                                                File "fill_between_demo.py", line 21, in <module>                             
    ax1.fill_between(x, 0, y1)                                                  
  File "/home/usr/.virtualenvs/raiju/lib/python3.6/site-packages/matplotlib/__init__.py", line 1898, in inner                                                  
    return func(ax, *args, **kwargs)                                            
  File "/home/usr/.virtualenvs/raiju/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 4778, in fill_between                                         
    x = ma.masked_invalid(self.convert_xunits(x))                               
  File "/home/usr/.virtualenvs/raiju/lib/python3.6/site-packages/numpy/ma/core.py", line 2388, in masked_invalid                                               
    condition = ~(np.isfinite(a))                                               
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''     

Why is that happening and how to fix it?

Note that plotting the data (aka not using fill*) works just fine.

Answer 1

The problem is, that the numpy ufunc isfinite is not defined for the numpy.datetime64 dtype. There is an effort to change this, though. This issue on numpy's github is being worked on in this pull-request, but as long as this is not finished up and merged, you will not be able to use isfinite on that dtype. This is a problem as matplotlib.pyplot.fill_between is using this function implicitly, when calling numpy.ma.masked_invalid to mask all invalid entries of your input array.

There is a work-around though. As pointed out in this answer to a similar question concerning fill_between plotting of a pandas Series of datetime64 type, pandas registers a custom converter for (among others) numpy arrays of datetime64 dtype with matplotlib. To make use of that, you simply have to import pandas:

import numpy as np
import matplotlib.pyplot as plt
import datetime
# import pandas for its converter that is then used in pyplot!
import pandas

x1 = np.arange(0.0, 2, 0.01)
now = np.datetime64(datetime.datetime.now())
x = np.array([now - np.timedelta64(datetime.timedelta(seconds=i))
              for i in range(200)])
y1 = np.sin(2 * np.pi * x1)
y2 = 1.2 * np.sin(4 * np.pi * x1)

fig, ax1 = plt.subplots(1, 1, sharex=True)

# Test support for masked arrays.
ax1.fill_between(x, 0, y1)
ax1.set_ylabel('between y1 and 0')
y2 = np.ma.masked_greater(y2, 1.0)
ax1.plot(x, y1, x, y2, color='black')
ax1.fill_between(
    x, y1, y2, where=y2 >= y1,
    facecolor='green',
    interpolate=True)
ax1.fill_between(x, y1, y2, where=y2 <= y1, facecolor='red', interpolate=True)
ax1.set_title('Now regions with y2>1 are masked')

# Show the plot.
plt.show()

will work and give you your desired output:

Answer 2

I had a similar issue with ax.setxlims, because I forgot to cast some datetime to np.datetime.

>>> import datetime
>>> import numpy
>>> a = datetime.datetime.now()
>>> numpy.isfinite(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
>>> a = numpy.datetime64(a)
>>> numpy.isfinite(a)
True

It looks like it is no longer required to import pandas