Add() function in tf.keras.Sequential()

Question

Is it possible to incorporate an Add() function in the tf.keras.Sequential() model, when defined like:

from tensorflow import keras

model = keras.Sequential([
    keras.Input(shape(input_shape,)),
    keras.layers.Dense(32),
    keras.layers.Dense(8),
    # I want to add here
    keras.layers.Add()(some_var)
], name='my_model')

some_var is a tensor of with the same size as the network at that point. So each element needs to be added to its corresponding element in some_var.

I know I can do this quite easily with the functional API, but would prefer to use a sequential model as it would match other branches in my network.

If its not clear keras.layers.Add()(some_var) is just a guess of how I would like it to work. This gives the error: ValueError: A merge layer should be called on a list of inputs..

My question is specific to the style in which I define the Sequential model.

Answer 1

One of the main difference between Functional and Sequential API is that Sequential works with single input and single output where as Functional API works with single-input and single-output or single-input and multiple-output, or multiple-inputs and multiple-outputs. So using Functional API, you can add two layers of multiple-inputs through `keras.layers.Add().

Also, this keras.layers.Add() can be used in to add two input tensors which is not really we do. we can rather use like d = tf.add(a,b). Both c and d are equal

a = tf.constant(1.,dtype=tf.float32, shape=(1,3)).  
b = tf.constant(2.,dtype=tf.float32, shape=(1,3)). 
c = tf.keras.layers.Add()([a, b]). 

The following example is from keras website. You can see how it is used in Functional API

import keras

input1 = keras.layers.Input(shape=(16,))
x1 = keras.layers.Dense(8, activation='relu')(input1)
input2 = keras.layers.Input(shape=(32,))
x2 = keras.layers.Dense(8, activation='relu')(input2)
# equivalent to added = keras.layers.add([x1, x2])
added = keras.layers.Add()([x1, x2])

out = keras.layers.Dense(4)(added)
model = keras.models.Model(inputs=[input1, input2], outputs=out)