numpy.where: TypeError: invalid type promotion

Question

_{I know there are other questions with the same error name, yet none of them match the np.where statement, and also I couldn't find the answer to my problem in them}

So I made a pandas DataFrame called data and created a Series out of it called dates, which is:

dates= pd.to_datetime(pd.to_timedelta(data.a_date, unit= 'D') + pd.datetime(1960,1,1), 
                      errors= 'coerse')

I need to clear some of the dates because they do not match with an indicator of them in data, so I tried to adjust that while keeping the indexes correct using numpy.where,
Yet I had gotten this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-2b83ed2b2468> in <module>()
----> 1 np.where(((dates.notnull()) & (data.a_IND == 0)), np.nan, dates)

TypeError: invalid type promotion

Answer 1

If you want to keep the date type, substitute np.nan with np.datetime64('NaT'):

np.where(((dates.notnull()) & (data.a_IND == 0)), np.datetime64('NaT'), dates)

Answer 2

The documentation of np.where(cond, x, y) says that the second and third arguments - x and y - need to be array or array_like. Also, I believe x and y must be of the same shape.

Your x is a scalar (np.nan) and y is an array_like object (dates). Maybe that's the problem.

Answer 3

I got a similar problem and managed to fix it by getting the date property from the index, i.e. this works:

np.where(condition, df.x, df.index.date)

And this doesn't work:

np.where(condition, df.x, df.index)

when the index has dtype='datetime64[ns]'

Hope that helps!