Avoid certain parameter combinations in GridSearchCV

Question

I'm using scikit-learn's GridSearchCV to iterate over a parameter space to tune a model. Specifically, I'm using it to test different hyperparameters in a neural network. The grid is as follows:

params = {'num_hidden_layers': [0,1,2],
          'hidden_layer_size': [64,128,256],
          'activation': ['sigmoid', 'relu', 'tanh']}

The problem is that I end up running redundant models when hidden num_hidden_layers is set to 0. It will run a model with 0 hidden layers and 64 units, another with 128 units, and another with 256 units. All of these models are equivalent since there is no hidden layer. This is highly inefficient and it means I need to write more code to remove redundancy in the results.

Is there a way to prevent such parameter combinations, perhaps by passing a tuple of parameters?

Answer 1

The sklearn documentation suggests two parameter grids.

So you could do something like this:

param_grid = [
    {'num_hidden_layers': [1,2],
      'hidden_layer_size': [64,128,256],
      'activation': ['sigmoid', 'relu', 'tanh']},
    {'num_hidden_layers': [0],
      'hidden_layer_size': [64],
      'activation': ['sigmoid', 'relu', 'tanh']}
    ]

Answer 2

GridSearchCV allows you to pass list of dictionaries to params:

param_grid : dict or list of dictionaries

Dictionary with parameters names (string) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.

So you can specify these dictionaries to be certain subdictionaries of your original dictionary. Thus, you could avoid irrelevant combinations.