BrokenProcessPool on using n_jobs parameter in cross_val_score

Question

I'm getting error when using n_jobs = -1 as a parameter in sklearn.model_selection.cross_val_score. I'm a beginner in Deep Learning and ANN and as per the instructor in this course in k-fold Cross-Validation, use n_jobs = -1 to use all the processors of CPU so as to reduce the time but it's throwing an error in my case.

Error -

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Full stack trace can be found here.

import keras
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
def build_classifier():
    classifier = Sequential()
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
    classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, nb_epoch = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = Y_train, cv = 10, n_jobs = -1)

Answer 1

Try creating your build_classifier function in an external file and importing it. E.g:

in file classifier_builder.py:

import keras

def build_classifier():
  classifier = Sequential()
  classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
  classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
  classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
  classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
return classifier

and then in your notebook:

import classifier_builder

classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, nb_epoch = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = Y_train, cv = 10, n_jobs = -1)

This solved the issue for me. Apparently, the inline function is not picklable.