How to implement hyperparameter search in Tensorflow?

Question

I would like to implement a hyperparameter search in Tensorflow, like the one presented in this video. Unfortunately I was not able to find any tutorials about it.

I've found some code which uses it, but I was not able to understand it properly. Implementing the bayesian optimization would be the best, but I would like to try the grid or random search first.

Should I create the different graphs before? How is it possible to do the training on multiple graphs, and how to compare them?

Answer 1

You can use DyTB (dynamic training bench): this tool allows you to focus only on the hyperparameter search, using tensorboard to compare the measured stats of the varisous trained model.

DyTB creates for you a unique name associated with the current set of hyperparameters and use it as a log dir. Creating different log directories allows the use of Tensorboard for an easy comparison.

For instance, you can train VGG on Cifar10 with this single line (both VGG and Cifar10 are some of predefined model & dataset available):

import tensorflow as tf
from dytb.inputs.predefined import Cifar10
from dytb.train import train
from dytb.models.predefined.VGG import VGG

# Instantiate the model
vgg = VGG()

# Instantiate the CIFAR-10 input source
cifar10 = Cifar10.Cifar10()

# 1: Train VGG on Cifar10 for 50 epochs
# Place the train process on GPU:0
device = '/gpu:0'
with tf.device(device):
    info = train(
        model=vgg,
        dataset=cifar10,
        hyperparameters={
            "epochs": 50,
            "batch_size": 50,
            "regularizations": {
                "l2": 1e-5,
                "augmentation": {
                    "name": "FlipLR",
                    "fn": tf.image.random_flip_left_right,
                    # factor is the estimated amount of augmentation
                    # that "fn" introduces.
                    # In this case, "fn" doubles the training set size
                    # Thus, an epoch is now seen as the original training
                    # training set size * 2
                    "factor": 2,
                }
            },
            "gd": {
                "optimizer": tf.train.AdamOptimizer,
                "args": {
                    "learning_rate": 1e-3,
                    "beta1": 0.9,
                    "beta2": 0.99,
                    "epsilon": 1e-8
                }
            }
        })

During the training of this model, you can monitor the loss trend and the accuracy value using tensorboard.

A new folder is created for you, using some of the representative hyperparameters used:

tensorboard --logdir "log/VGG/CIFAR-10_Adam_l2=1e-05_fliplr/"

as you can see, a folder for the model is created and the hyperparameter used to train it are added next, as subfolder.

This means that if you change the Optimizer (from ADAM to MomentumOptimizer) or you add a comment, or you change the l2 regularization parameter, ecc, DyTB creates a subfolder into the VGG folder.

This allow you to compare the measured metric with tensorboard, using the model directory as logdir, in this way:

tensorboard --logdir log/VGG

For a more comprehensive guide, just have a look at the DyTB README.md or the python-notebook example.

Answer 2

Another viable option for grid search with Tensorflow is Tune. It's a scalable framework/tool for hyperparameter tuning, specifically for deep learning/reinforcement learning.

It also takes care of Tensorboard logging and efficient search algorithms (ie, HyperOpt integration and HyperBand) in about 10 lines of Python.

import ray
from ray import tune

def train_tf_model(config):
    model = Model(lr=config["lr"])
    for x, y in dataset:
        accuracy = model.fit(x, y)
        tune.track.log(accuracy=accuracy)

tune.run(
    train_tf_model, 
    config={"lr": tune.grid_search([0.2, 0.4, 0.6])}
)

(Disclaimer: I contribute actively to this project!)