passing keras optimizer as string parameter to keras optimizer function

Question

I’m tuning hyperparameters of a keras deep learning model with the help of a config.json file containing hyperparameters.

    { “opt: “Adam”,
      “lr”: 0.01,
       “grad_clip”: 0.5
    }

Keras allows to specify an optimizer in two ways:

1. As string argument in call to function without additional parameters.

model.compile(loss='categorical_crossentropy’,
              optimizer=’Adam’, 
              metrics=['mse'])

2. As eponymous function with additional parameters.

model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.Adam(lr=0.01, clipvalue=0.5), 
              metrics=['mse'])

My question is: how to pass the optimizer (SGD, Adam etc.) as argument from config file along with the subparameters and employ the keras.optimizers.optimizer() function call as in (2)?

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed, Bidirectional
from keras import optimizers

def train(X,y, opt, lr, clip):

   model = Sequential()
   model.add(Bidirectional(LSTM(100, return_sequences=True), input_shape=(500, 300)))    
   model.add(TimeDistributed(Dense(5, activation='sigmoid')))

   model.compile(loss='categorical_crossentropy',
                  optimizer=optimizers.opt(lr=lr, clipvalue=clip), 
                  metrics=['mse'])

   model.fit(X, y, epochs=100, batch_size=1, verbose=2)

   return(model)

When I try to pass parameters from my config file to the above train() function, I get the following error:

AttributeError: module 'keras.optimizers' has no attribute 'opt'

How do I parse the optimizer in string from as a function?

Answer 1

At least for tensorflow version 2.11 there is a function to get the optimizer by its name and hand over parameters. Check this for further information: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/get.

To make it short, here is a minimal example for its usage:

# read in your config.json file
# KEYS NEED TO BE ADJUSTED TO MAKE THIS WORK CORRECTLY:
# { 
#   “identifier: “Adam”,
#   “learning_rate”: 0.01,
#   “grad_clip”: 0.5
# }

import tensorflow as tf
import json

with open('path/to/config.json', 'r') as f:
    opt_conf = json.load(f)


# define the model
model = tf.keras.Sequential()

# then create the optimizer using the data from opt_conf
optimizer = tf.keras.optimizers.get(**opt_conf)

# compile the model
model.compile(loss='categorical_crossentropy',
              optimizer=optimizer, 
              metrics=['mse'])