How to deal with NaN value when plot boxplot using python

Question

I am using matplotlib to plot a box figure but there are some missing values (NaN). Then I found it doesn't display the box figure within the columns having NaN values. Do you know how to solve this problem? Here are the codes.

import numpy as np
import matplotlib.pyplot as plt

#==============================================================================
# open data
#==============================================================================
filename='C:\\Users\\liren\\OneDrive\\Data\\DATA in the first field-final\\ks.csv'

AllData=np.genfromtxt(filename,delimiter=";",skip_header=0,dtype='str')

TreatmentCode = AllData[1:,0]
RepCode = AllData[1:,1]
KsData= AllData[1:,2:].astype('float')
DepthHeader = AllData[0,2:].astype('float')
TreatmentUnique = np.unique(TreatmentCode)[[3,1,4,2,8,6,9,7,0,5,10],]
nT = TreatmentUnique.size#nT=number of treatments
#nD=number of deepth;nR=numbers of replications;nT=number of treatments;iT=iterms of treatments
nD = 5
nR = 6
KsData_3D = np.zeros((nT,nD,nR)) 

for iT in range(nT):
    Treatment = TreatmentUnique[iT]

    TreatmentFilter = TreatmentCode == Treatment

    KsData_Filtered = KsData[TreatmentFilter,:]
    
    KsData_3D[iT,:,:] = KsData_Filtered.transpose()iD = 4
                      
fig=plt.figure()
ax = fig.add_subplot(111)
plt.boxplot(KsData_3D[:,iD,:].transpose())
ax.set_xticks(range(1,nT+1))
ax.set_xticklabels(TreatmentUnique)
ax.set_title(DepthHeader[iD])

Here is the final figure and some of the treatments are missing in the box.

Answer 1

You can remove the NaNs from the data first, then plot the filtered data.

To do that, you can first find the NaNs using np.isnan(data), then perform the bitwise inversion of that boolean array using the ~ operator. Use that to index the data array, and you filter out the NaNs.

filtered_data = data[~np.isnan(data)]

In a complete example (adapted from here)

For 1D data:

import matplotlib.pyplot as plt
import numpy as np

# fake up some data
spread = np.random.rand(50) * 100
center = np.ones(25) * 50
flier_high = np.random.rand(10) * 100 + 100
flier_low = np.random.rand(10) * -100
data = np.concatenate((spread, center, flier_high, flier_low), 0)

# Add a NaN
data[40] = np.NaN

# Filter data using np.isnan
filtered_data = data[~np.isnan(data)]

# basic plot
plt.boxplot(filtered_data)

plt.show()

For 2D data:

For 2D data, you can't simply use the mask above, since then each column of the data array would have a different length. Instead, we can create a list, with each item in the list being the filtered data for each column of the data array.

A list comprehension can do this in one line: [d[m] for d, m in zip(data.T, mask.T)]

import matplotlib.pyplot as plt
import numpy as np

# fake up some data
spread = np.random.rand(50) * 100
center = np.ones(25) * 50
flier_high = np.random.rand(10) * 100 + 100
flier_low = np.random.rand(10) * -100
data = np.concatenate((spread, center, flier_high, flier_low), 0)

data = np.column_stack((data, data * 2., data + 20.))

# Add a NaN
data[30, 0] = np.NaN
data[20, 1] = np.NaN

# Filter data using np.isnan
mask = ~np.isnan(data)
filtered_data = [d[m] for d, m in zip(data.T, mask.T)]

# basic plot
plt.boxplot(filtered_data)

plt.show()

I'll leave it as an exercise to the reader to extend this to 3 or more dimensions, but you get the idea.