Polars dataframe drop nans

Question

I need to drop rows that have a nan value in any column. As for null values with drop_nulls()

df.drop_nulls()

but for nans. I have found that the method drop_nans exist for Series but not for DataFrames

df['A'].drop_nans()

Pandas code that I'm using:

df = pd.DataFrame(
    {
        'A': [0, 0, 0, 1,None, 1],
        'B': [1, 2, 2, 1,1, np.nan]
    }
)
df.dropna()

Answer 1

Update: Polars 1.16.0 added a dedicated .drop_nans() method.

df.drop_nans()

shape: (4, 3)
┌─────┬─────┬─────┐
│ A   ┆ B   ┆ C   │
│ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ str │
╞═════╪═════╪═════╡
│ 0.0 ┆ 1.0 ┆ a   │
│ 0.0 ┆ 2.0 ┆ b   │
│ 0.0 ┆ 2.0 ┆ c   │
│ 1.0 ┆ 1.0 ┆ d   │
└─────┴─────┴─────┘

---

Original answer

Another definition could be: keep rows where all values are not NaN

For that we could use:

• .is_not_nan() to test for "not nan"
• pl.col(pl.Float32, pl.Float64) to select only float columns
• .all_horizontal() to compute a row-wise True/False comparison
• DataFrame.filter to keep only the "True" rows

df = pl.from_repr("""
┌─────┬─────┬─────┐
│ A   ┆ B   ┆ C   │
│ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ str │
╞═════╪═════╪═════╡
│ 0.0 ┆ 1.0 ┆ a   │
│ 0.0 ┆ 2.0 ┆ b   │
│ 0.0 ┆ 2.0 ┆ c   │
│ 1.0 ┆ 1.0 ┆ d   │
│ NaN ┆ 1.0 ┆ e   │
│ 1.0 ┆ NaN ┆ g   │
└─────┴─────┴─────┘
""")

df.filter(
    pl.all_horizontal(pl.col(pl.Float32, pl.Float64).is_not_nan())
)

shape: (4, 3)
┌─────┬─────┬─────┐
│ A   ┆ B   ┆ C   │
│ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ str │
╞═════╪═════╪═════╡
│ 0.0 ┆ 1.0 ┆ a   │
│ 0.0 ┆ 2.0 ┆ b   │
│ 0.0 ┆ 2.0 ┆ c   │
│ 1.0 ┆ 1.0 ┆ d   │
└─────┴─────┴─────┘

The Selectors API provides additional helpers e.g. cs.float()

df.filter(
    pl.all_horizontal(cs.float().is_not_nan())
)