Making scikit deterministic?

Question

I'm using scikit-learn to train some classifiers. I do cross validation and then compute AUC. However I'm getting a different AUC number every time I run the tests although I made sure to use a seed and a RandomState. I want my tests to be deterministic. Here's my code:

from sklearn.utils import shuffle
SEED = 0
random_state = np.random.RandomState(SEED)
X, y = shuffle(data, Y, random_state=random_state)
X_train, X_test, y_train, y_test = \
        cross_validation.train_test_split(X, y, test_size=test_size, random_state=random_state)
clf = linear_model.LogisticRegression()
kfold = cross_validation.KFold(len(X), n_folds=n_folds)
mean_tpr = 0.0
mean_fpr = np.linspace(0, 1, 100)

for train, test in kfold:
        probas_ = clf.fit(X[train], Y[train]).predict_proba(X[test])
        fpr, tpr, thresholds = roc_curve(Y[test], probas_[:, 1])
        mean_tpr += interp(mean_fpr, fpr, tpr)
        mean_tpr[0] = 0.0

mean_tpr /= len(kfold)
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)

My questions: 1- Is there something wrong in my code that's making the results different each time I run it? 2- Is there a global way to make scikit deterministic?

EDIT:

I just tried this:

test_size = 0.5
X = np.random.randint(10, size=(10,2))
Y = np.random.randint(2, size=(10))
SEED = 0
random_state = np.random.RandomState(SEED)
X_train, X_test, y_train, y_test = \
    cross_validation.train_test_split(X, Y, test_size=test_size, random_state=random_state)

print X_train # I recorded the result

Then I did:

X_train, X_test, y_train, y_test = \
    cross_validation.train_test_split(X, Y, test_size=test_size, random_state=6) #notice the change in random_state

Then I did:

X_train, X_test, y_train, y_test = \
    cross_validation.train_test_split(X, Y, test_size=test_size, random_state=random_state)

print X_train #the result is different from the first one!!!!

As you see I'm getting different results although I used the same random_state! How to solve this?

Answer 1

LogisticRegression uses randomness internally and has an (undocumented, will fix in a moment) random_state argument.

There's no global way of setting the random state, because unfortunately the random state on LogisticRegression and the SVM code can only be set in a hacky way. That's because this code comes from Liblinear and LibSVM, which use the C standard library's rand function and that cannot be seeded in a principled way.

EDIT The above is true, but probably not the cause of the problem. You're threading a single np.random.RandomState through your calls, while you should pass the same integer seed for easy reproducibility.