Hyperparameter optimization for Pytorch model [closed]

Question

What is the best way to perform hyperparameter optimization for a Pytorch model? Implement e.g. Random Search myself? Use Skicit Learn? Or is there anything else I am not aware of?

Answer 1

Many researchers use RayTune. It's a scalable hyperparameter tuning framework, specifically for deep learning. You can easily use it with any deep learning framework (2 lines of code below), and it provides most state-of-the-art algorithms, including HyperBand, Population-based Training, Bayesian Optimization, and BOHB.

import torch.optim as optim from ray import tune from ray.tune.examples.mnist_pytorch import get_data_loaders, ConvNet, train, test   def train_mnist(config):     train_loader, test_loader = get_data_loaders()     model = ConvNet()     optimizer = optim.SGD(model.parameters(), lr=config["lr"])     for i in range(10):         train(model, optimizer, train_loader)         acc = test(model, test_loader)         tune.report(mean_accuracy=acc)   analysis = tune.run(     train_mnist, config={"lr": tune.grid_search([0.001, 0.01, 0.1])})  print("Best config: ", analysis.get_best_config(metric="mean_accuracy"))  # Get a dataframe for analyzing trial results. df = analysis.dataframe() 

[Disclaimer: I contribute actively to this project!]

Answer 2

What I found is following:

• HyperOpt
• Optuna
• tune

More young projects:

• hypersearch limited only to FC layers.
• skorch Just grid search available
• Auto-PyTorch

UPDATE something new:

• Ax: Adaptive Experimentation Platform by facebook

• BoTorch: Bayesian Optimization in PyTorch

Also, I found a useful table at post by @Richard Liaw: