Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I add outlier detection and removal to Scikit learn Pipeline?

I want to create a Pipeline in Scikit-Learn with a specific step being outlier detection and removal, allowing the transformed data to be passed to other transformers and estimator.

I have searched SE but can't find this answer anywhere. Is this possible?

like image 662
Attack68 Avatar asked Sep 15 '18 16:09

Attack68


People also ask

How do you remove outliers from a pipeline?

The imblearn package provides its own set of samplers, but we can also use a custom sampler with imblearn. FunctionSampler. We can take advantage of those features to create our function to remove outliers and call it within the pipeline as a sampler.

What is the difference between anomaly detection and novelty detection?

In simple terms, we can think of anomalies as unusual or unexpected data instances within a dataset. The term is often used interchangeably with outliers. Similarly, novelties are also anomalies in data, but they only exist in new instances.

What is the benefit of using scikit-learn pipeline utility for data pre processing?

The Scikit-learn pipeline is a tool that chains all steps of the workflow together for a more streamlined procedure. The key benefit of building a pipeline is improved readability. Pipelines are able to execute a series of transformations with one call, allowing users to attain results with less code.


1 Answers

Yes. Subclass the TransformerMixin and build a custom transformer. Here is an extension to one of the existing outlier detection methods:

from sklearn.pipeline import Pipeline, TransformerMixin
from sklearn.neighbors import LocalOutlierFactor

class OutlierExtractor(TransformerMixin):
    def __init__(self, **kwargs):
        """
        Create a transformer to remove outliers. A threshold is set for selection
        criteria, and further arguments are passed to the LocalOutlierFactor class

        Keyword Args:
            neg_conf_val (float): The threshold for excluding samples with a lower
               negative outlier factor.

        Returns:
            object: to be used as a transformer method as part of Pipeline()
        """

        self.threshold = kwargs.pop('neg_conf_val', -10.0)

        self.kwargs = kwargs

    def transform(self, X, y):
        """
        Uses LocalOutlierFactor class to subselect data based on some threshold

        Returns:
            ndarray: subsampled data

        Notes:
            X should be of shape (n_samples, n_features)
        """
        X = np.asarray(X)
        y = np.asarray(y)
        lcf = LocalOutlierFactor(**self.kwargs)
        lcf.fit(X)
        return (X[lcf.negative_outlier_factor_ > self.threshold, :],
                y[lcf.negative_outlier_factor_ > self.threshold])

    def fit(self, *args, **kwargs):
        return self

Then create a pipeline as:

pipe = Pipeline([('outliers', OutlierExtraction()), ...])
like image 132
Attack68 Avatar answered Oct 22 '22 10:10

Attack68