Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use Isolation Forest

Tags:

I am trying to detect the outliers to my dataset and I find the sklearn's Isolation Forest. I can't understand how to work with it. I fit my training data in it and it gives me back a vector with -1 and 1 values.

Can anyone explain to me how it works and provide an example?

How can I know that the outliers are 'real' outliers?

Tuning Parameters?

Here is my code:

clf = IsolationForest(max_samples=10000, random_state=10) clf.fit(x_train) y_pred_train = clf.predict(x_train) y_pred_test = clf.predict(x_test)  [1 1 1 ..., -1 1 1] 
like image 303
dapo Avatar asked Mar 28 '17 07:03

dapo


People also ask

How does Isolation Forest work?

In an Isolation Forest, randomly sub-sampled data is processed in a tree structure based on randomly selected features. The samples that travel deeper into the tree are less likely to be anomalies as they required more cuts to isolate them.

Why is Isolation Forest better?

Isolation Forest is the best Anomaly Detection Algorithm for Big Data Right Now. Isolation forest or “iForest” is an astoundingly beautiful and elegantly simple algorithm that identifies anomalies with few parameters. The original paper is accessible to a broad audience and contains minimal math.

What is Isolation Forest in Python?

Isolation forest is a machine learning algorithm for anomaly detection. It's an unsupervised learning algorithm that identifies anomaly by isolating outliers in the data. Isolation Forest is based on the Decision Tree algorithm.

What type of algorithm is Isolation Forest?

Isolation forest is an anomaly detection algorithm. It detects anomalies using isolation (how far a data point is to the rest of the data), rather than modelling the normal points.


1 Answers

It seems you have many questions, let me try to answer them one by one to the best of my knowledge.

How it works?

It works due to the fact that the nature of outliers in any data set, which is outliers, is few and different, which is quite different from the typical clustering-based or distance-based algorithm. At the top level, it works on the logic that outliers take fewer steps to 'isolate' compare to the 'normal' point in any data set. To do so, this is what IF does; suppose you have training data set X with n data points, each having m features. In training, IF creates Isolation trees (Binary search trees) for different features.

For training, you have 3 parameters for tuning during the train phase:

  1. number of isolation trees (n_estimators in sklearn_IsolationForest)
  2. number of samples (max_samples in sklearn_IsolationForest)
  3. number of features to draw from X to train each base estimator (max_features in sklearn_IF).

max_samples is the number of random samples it will pick from the original data set for creating Isolation trees.

During the test phase:

  • sklearn_IF finds the path length of data point under test from all the trained Isolation Trees and finds the average path length. The higher the path length, the more normal the point, and vice-versa.

  • Based on the average path length. It calculates the anomaly score, decision_function of sklearn_IF can be used to get this. For sklearn_IF, the lower the score, the more anomalous the sample.

  • Based on the anomaly score, you can decide whether the given sample is anomalous or not by setting the proper value of contamination in the sklearn_IF object. The default value of contamination is 0.1, which you can tune for deciding the threshold. The amount of contamination of the data set, i.e., the proportion of outliers in the data set.

Tuning parameters

Training -> n_estimators, max_samples, max_features.

Testing -> contamination

like image 113
Bhargav Upadhyay Avatar answered Oct 18 '22 12:10

Bhargav Upadhyay