f1 score of all classes from scikits cross_val_score

Question

I'm using cross_val_score from scikit-learn (package sklearn.cross_validation) to evaluate my classifiers.
If I use f1 for the scoring parameter, the function will return the f1-score for one class. To get the average I can use f1_weighted but I can't find out how to get the f1-score of the other class. (precision and recall analogous)

The functions in sklearn.metrics have a labels parameter which does this, but I can't find anything like this in the documentation.

Is there a way to get the f1-score for all classes at once or at least specify the class which should be considered with cross_val_score?

Answer 1

When you create a scorer with make_scorer function you can pass any additional arguments you need, like this:

cross_val_score(
    svm.SVC(kernel='rbf', gamma=0.7, C = 1.0),
    X, y,
    scoring=make_scorer(f1_score, average='weighted', labels=[2]),
    cv=10)

But cross_val_score only allows you to return one score. You can't get scores for all classes at once without additional tricks. If you need that please refer to another stack overflow question which covers exactly that: Evaluate multiple scores on sklearn cross_val_score