handling missing data in Pandas

Question

I have a 200,000 x 500 dataframe loaded into Pandas. Is there a function that can automatically tell me which columns are missing data? Or do I have to iterate over each column and check element by element?

Once I've found a missing element, how do I define a custom function (based on both the column name and some other data in the same row) to do automatic replacements. I see the fillna() method, but I don't think it takes a (lambda) function as an input.

Thanks!

Answer 1

something like:

import pandas as pd
pd.isnull(frame).any()

Is probably what you're looking for to look for missing data

fillna currently does not take lambda functions though that's in the works as an open issue on github.

You can use DataFrame.apply to do custom filling for now. Though can you be a little more specific on what you need to do to fill the data? Just curious what the use case is.