Multiple inputs of keras model with tf.data.Dataset.from_generator in Tensorflow 2

Question

I am trying to implement a model in keras that will have multiple inputs:

• image (200x200)
• some numbers (1x50)
• three 1d signals (1x50000, 2x100000)

To feed that model, I want to write a generator to use with tf.data.Dataset.from_generator. From the docs of from_generator, its not clear to me how I should provide its parameters output_types, output_shapes. Can anyone help me with this?

Answer 1

I had a similar issue, and it took me many tries to get the structure right for those inputs. Here's an example of a network with 3 inputs and 2 outputs, complete to the .fit call.

The following works in tensorflow 2.1.0

import tensorflow as tf
import numpy as np

def generator(N=10):
    """
    Returns tuple of (inputs,outputs) where
    inputs  = (inp1,inp2,inp2)
    outputs = (out1,out2)
    """
    dt=np.float32
    for i in range(N):
        inputs  = (np.random.rand(N,3,3,1).astype(dt), 
                   np.random.rand(N,3,3,1).astype(dt), 
                   np.random.rand(N,3,3,1).astype(dt))
        outputs = (np.random.rand(N,3,3,1).astype(dt),
                   np.random.rand(N,3,3,1).astype(dt))
        yield inputs,outputs

# Create dataset from generator
types = ( (tf.float32,tf.float32,tf.float32),
          (tf.float32,tf.float32) )
shapes = (([None,3,3,1],[None,3,3,1],[None,3,3,1]),
          ([None,3,3,1],[None,3,3,1]))
data = tf.data.Dataset.from_generator(generator,
                                      output_types=types,
                                      output_shapes=shapes
                                     )
# Define a model
inp1 = tf.keras.Input(shape=(3,3,1),name='inp1')
inp2 = tf.keras.Input(shape=(3,3,1),name='inp2')
inp3 = tf.keras.Input(shape=(3,3,1),name='inp3')
out1 = tf.keras.layers.Conv2D(1,kernel_size=3,padding='same')(inp1)
out2 = tf.keras.layers.Conv2D(1,kernel_size=3,padding='same')(inp2)
model = tf.keras.Model(inputs=[inp1,inp2,inp3],outputs=[out1,out2])
model.compile(loss=['mse','mse'])

# Train
model.fit(data)