Scikit Learn SVC decision_function and predict

Question

I'm trying to understand the relationship between decision_function and predict, which are instance methods of SVC (http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html). So far I've gathered that decision function returns pairwise scores between classes. I was under the impression that predict chooses the class that maximizes its pairwise score, but I tested this out and got different results. Here's the code I was using to try and understand the relationship between the two. First I generated the pairwise score matrix, and then I printed out the class that has maximal pairwise score which was different than the class predicted by clf.predict.

        result = clf.decision_function(vector)[0]         counter = 0         num_classes = len(clf.classes_)         pairwise_scores = np.zeros((num_classes, num_classes))         for r in xrange(num_classes):             for j in xrange(r + 1, num_classes):                 pairwise_scores[r][j] = result[counter]                 pairwise_scores[j][r] = -result[counter]                 counter += 1          index = np.argmax(pairwise_scores)         class = index_star / num_classes         print class         print clf.predict(vector)[0] 

Does anyone know the relationship between these predict and decision_function?

Answer 1

I don't fully understand your code, but let's go trough the example of the documentation page you referenced:

import numpy as np X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]]) y = np.array([1, 1, 2, 2]) from sklearn.svm import SVC clf = SVC() clf.fit(X, y)  

Now let's apply both the decision function and predict to the samples:

clf.decision_function(X) clf.predict(X) 

The output we get is:

array([[-1.00052254],        [-1.00006594],        [ 1.00029424],        [ 1.00029424]]) array([1, 1, 2, 2]) 

And that is easy to interpret: The desion function tells us on which side of the hyperplane generated by the classifier we are (and how far we are away from it). Based on that information, the estimator then label the examples with the corresponding label.

Answer 2

For those interested, I'll post a quick example of the predict function translated from C++ (here) to python:

# I've only implemented the linear and rbf kernels def kernel(params, sv, X):     if params.kernel == 'linear':         return [np.dot(vi, X) for vi in sv]     elif params.kernel == 'rbf':         return [math.exp(-params.gamma * np.dot(vi - X, vi - X)) for vi in sv]  # This replicates clf.decision_function(X) def decision_function(params, sv, nv, a, b, X):     # calculate the kernels     k = kernel(params, sv, X)      # define the start and end index for support vectors for each class     start = [sum(nv[:i]) for i in range(len(nv))]     end = [start[i] + nv[i] for i in range(len(nv))]      # calculate: sum(a_p * k(x_p, x)) between every 2 classes     c = [ sum(a[ i ][p] * k[p] for p in range(start[j], end[j])) +           sum(a[j-1][p] * k[p] for p in range(start[i], end[i]))                 for i in range(len(nv)) for j in range(i+1,len(nv))]      # add the intercept     return [sum(x) for x in zip(c, b)]  # This replicates clf.predict(X) def predict(params, sv, nv, a, b, cs, X):     ''' params = model parameters         sv = support vectors         nv = # of support vectors per class         a  = dual coefficients         b  = intercepts          cs = list of class names         X  = feature to predict            '''     decision = decision_function(params, sv, nv, a, b, X)     votes = [(i if decision[p] > 0 else j) for p,(i,j) in enumerate((i,j)                                             for i in range(len(cs))                                            for j in range(i+1,len(cs)))]      return cs[max(set(votes), key=votes.count)] 

There are a lot of input arguments for predict and decision_function, but note that these are all used internally in by the model when calling predict(X). In fact, all of the arguments are accessible to you inside the model after fitting:

# Create model clf = svm.SVC(gamma=0.001, C=100.)  # Fit model using features, X, and labels, Y. clf.fit(X, y)  # Get parameters from model params = clf.get_params() sv = clf.support_vectors nv = clf.n_support_ a  = clf.dual_coef_ b  = clf._intercept_ cs = clf.classes_  # Use the functions to predict print(predict(params, sv, nv, a, b, cs, X))  # Compare with the builtin predict print(clf.predict(X))