Parameter oob_score_ in scikit-learn equals accuracy or error?

Question

I implemented Random Forest classifiers (RF) from Python scikit-learn package for a ML problem. In the first stage I used cross validation to spot check other algorithms and RF is now my choice.

Later on I also checked out what the OOB estimation of RF tells me. However, when I compare the return in 'oob_score_' with my results from CV I have a large discrepancy.

The scikit-learn doc tells me:

oob_score : bool

Whether to use out-of-bag samples to estimate the generalization error.

Because of the doc I was assuming that the Parameter 'oob_score_' is the error estimation. But looking for reasons it also came to my mind that it might actually estimate the accuracy instead This would be - at least a bit - closer to my CV results. I checked also the code, and more believe it's the accuracy but wanted to be sure... (in this case I find the doc misleading BTW).

Is oob_score_ in scikit-learn accuracy or error estimation?

Answer 1

It is an analogous of .score method, which returns accuracy of the model. It simply generalizes to to the oob scenario. Documentation is indeed a bit missleading.

As you may find in the code https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/forest.py

for k in range(self.n_outputs_):
            if (predictions[k].sum(axis=1) == 0).any():
                warn("Some inputs do not have OOB scores. "
                     "This probably means too few trees were used "
                     "to compute any reliable oob estimates.")

            decision = (predictions[k] /
                        predictions[k].sum(axis=1)[:, np.newaxis])
            oob_decision_function.append(decision)
            oob_score += np.mean(y[:, k] ==
                                 np.argmax(predictions[k], axis=1), axis=0)

It simply computes average of correct classifications.