Pandera validate get all valid rows

Question

I am trying to use pandera library (I am very new with this) for pandas dataframe validation. What I want to do is to ignore the rows which are not valid as per the schema. How can I do that?

for example: pandera schema looks like below:

import pandera as pa
import pandas as pd

schema: pa.DataFrameSchema = pa.DataFrameSchema(columns={
  'Col1': pa.Column(str),
  'Col2': pa.Column(float, checks=pa.Check(lambda x: (0 <= x <= 1)), nullable=True),
})

df: pd.DataFrame = pd.DataFrame({
    "Col1": ["1", "2", "3", nan],
    "Col2": [0.3, 0.4, 5, 0.2],
})

What I want to do is when I apply validation on the df I get a result:

   Col1  Col2
0     1   0.3
1     2   0.4

The other rows with error dropped.

Answer 1

pandera author here!

Currently you have to use a try except block with lazy validation. The SchemaErrors.failure_cases df doesn't always have an index in certain cases, like if the column's type is incorrect. The index only applies to checks that produce an index-aligned boolean dataframe/series.

By default the check_fn function fed into pa.Check should take a pandas Series as input. I fixed your custom check like so:

import pandera as pa
import pandas as pd
import numpy as np

schema: pa.DataFrameSchema = pa.DataFrameSchema(columns={
  'Col1': pa.Column(str),
  'Col2': pa.Column(
      float, checks=pa.Check(lambda series: series.between(0, 1)), nullable=True
    ),
})

df: pd.DataFrame = pd.DataFrame({
    "Col1": ["1", "2", "3", np.nan],
    "Col2": [0.3, 0.4, 5, 0.2],
})

try:
    schema(df, lazy=True)
except pa.errors.SchemaErrors as exc:
    filtered_df = df[~df.index.isin(exc.failure_cases["index"])]

print(f"filtered df:\n{filtered_df}")

Output:

filtered df:
  Col1  Col2
0    1   0.3
1    2   0.4

To check value ranges I'd recommend using the built-in pa.Check.in_range check.

In other cases, just be aware of the element_wise=True kwarg in pa.Check, it modifies the expected type signature of the check_fn arg.