Pandas: Remove all NaN values in all columns [duplicate]

Question

I have a data frame with many null records:

Col_1    Col_2      Col_3
10         5          2
22         7          7
3         9          5       
4         NaN       NaN
5         NaN       NaN
6         4         NaN
7         6          7
8         10        NaN
12        NaN        1

I want to remove all NaN values in all rows of columns . As you could see, each column has different number of rows. So, I want to get something like this:

Col_1    Col_2      Col_3
10         5          2
22         7          7
3          9          5       
4          4          7
6          6          1
7         10          
8                 
12    

I tried

filtered_df = df.dropna(how='any')

But it removes all records in the dataframe. How may I do that ?

Answer 1

You can also use pd.concat on a list of series.

Note that columns Col_2 and Col_3 are unavoidably float due to NaN elements, if you remove dtype=object as an option.

res = pd.concat([df[x].dropna().reset_index(drop=True) for x in df], axis=1)

print(res)

   Col_1  Col_2  Col_3
0     10    5.0    2.0
1     22    7.0    7.0
2      3    9.0    5.0
3      4    4.0    7.0
4      5    6.0    1.0
5      6   10.0    NaN
6      7    NaN    NaN
7      8    NaN    NaN
8     12    NaN    NaN