Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hyperparameter Tuning of Tensorflow Model

Tags:

I've used Scikit-learn's GridSearchCV before to optimize the hyperparameters of my models, but just wondering if a similar tool exists to optimize hyperparameters for Tensorflow (for instance number of epochs, learning rate, sliding window size etc.)

And if not, how can I implement a snippet that effectively runs all different combinations?

like image 898
mamafoku Avatar asked Jun 28 '17 12:06

mamafoku


People also ask

How do you do a hyperparameter tuning in deep learning?

The hyperparameters to tune are the number of neurons, activation function, optimizer, learning rate, batch size, and epochs. The second step is to tune the number of layers. This is what other conventional algorithms do not have. Different layers can affect the accuracy.

What is model hyperparameter tuning?

Hyperparameter tuning consists of finding a set of optimal hyperparameter values for a learning algorithm while applying this optimized algorithm to any data set. That combination of hyperparameters maximizes the model's performance, minimizing a predefined loss function to produce better results with fewer errors.


2 Answers

Even though it does not seem to be explicitly documented (in version 1.2), the package tf.contrib.learn (included in TensorFlow) defines classifiers that are supposed to be compatible with scikit-learn... However, looking at the source, it seems you need to explicitly set the environment variable TENSORFLOW_SKLEARN (e.g. to "1") to actually get this compatibility. If this works, you can already use GridSearchCV (see this test case).

That said, there are a few alternatives. I don't know about any specific to TensorFlow, but hyperopt, Scikit-Optimize or SMAC3 should all be valid options. MOE and Spearmint look like used to be good choices but now don't seem too maintained.

Alternatively, you can look into a service like SigOpt (a company by the original author of MOE).

Edit

About running all possible combinations of parameters, the core logic, if you want to implement it yourself, is not really complicated. You can just define lists with the possible values for each parameter and then run through all the combinations with itertools.product. Something like:

from itertools import product

param1_values = [...]
param2_values = [...]
param3_values = [...]
for param1, param2, param3 in product(param1_values, param2_values param3_values):
    run_experiment(param1, param2, param3)

Note however that grid search can be prohibitively expensive to run in many cases, and even doing just a random search in the parameters space will probably be more efficient (more about that in this publication).

like image 159
jdehesa Avatar answered Sep 30 '22 16:09

jdehesa


Another viable (and documented) option for grid search with Tensorflow is Ray Tune. It's a scalable framework for hyperparameter tuning, specifically for deep learning/reinforcement learning.

You can try out a fast tutorial here.

It also takes care of Tensorboard logging and efficient search algorithms (ie, HyperOpt integration and HyperBand) in about 10 lines of Python.

from ray import tune

def train_tf_model(config):  
    for i in range(num_epochs):
        accuracy = train_one_epoch(model)
        tune.report(acc=accuracy)

tune.run(train_tf_model,
         config={
            "alpha": tune.grid_search([0.2, 0.4, 0.6]),
            "beta": tune.grid_search([1, 2]),
         })

(Disclaimer: I contribute actively to this project!)

like image 41
richliaw Avatar answered Sep 30 '22 18:09

richliaw