How to use fit_generator with multiple inputs

Question

Is it possible to have two fit_generator?

I'm creating a model with two inputs, The model configuration is shown below.

Label Y uses the same labeling for X1 and X2 data.

The following error will continue to occur.

Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[0.75686276, 0.75686276, 0.75686276], [0.75686276, 0.75686276, 0.75686276], [0.75686276, 0.75686276, 0.75686276], ..., [0.65882355, 0.65882355, 0.65882355...

My code looks like this:

def generator_two_img(X1, X2, Y,batch_size):
    generator = ImageDataGenerator(rotation_range=15,
                                   width_shift_range=0.2,
                                   height_shift_range=0.2,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True,
                                   fill_mode='nearest')

    genX1 = generator.flow(X1, Y, batch_size=batch_size)
    genX2 = generator.flow(X2, Y, batch_size=batch_size)

    while True:
        X1 = genX1.__next__()
        X2 = genX2.__next__()
        yield [X1, X2], Y
  """
      .................................
  """
hist = model.fit_generator(generator_two_img(x_train, x_train_landmark, 
                y_train, batch_size),
                steps_per_epoch=len(x_train) // batch_size, epochs=nb_epoch,
                callbacks = callbacks,
                validation_data=(x_validation, y_validation),
                validation_steps=x_validation.shape[0] // batch_size, 
                `enter code here`verbose=1)

Answer 1

Try this generator:

def generator_two_img(X1, X2, y, batch_size):     genX1 = gen.flow(X1, y,  batch_size=batch_size, seed=1)     genX2 = gen.flow(X2, y, batch_size=batch_size, seed=1)     while True:         X1i = genX1.next()         X2i = genX2.next()         yield [X1i[0], X2i[0]], X1i[1] 

Generator for 3 inputs:

def generator_three_img(X1, X2, X3, y, batch_size):     genX1 = gen.flow(X1, y,  batch_size=batch_size, seed=1)     genX2 = gen.flow(X2, y, batch_size=batch_size, seed=1)     genX3 = gen.flow(X3, y, batch_size=batch_size, seed=1)     while True:         X1i = genX1.next()         X2i = genX2.next()         X3i = genX3.next()         yield [X1i[0], X2i[0], X3i[0]], X1i[1] 

EDIT (add generator, output image and numpy array, and target)

#X1 is an image, y is the target, X2 is a numpy array - other data input         def gen_flow_for_two_inputs(X1, X2, y):     genX1 = gen.flow(X1,y,  batch_size=batch_size,seed=666)     genX2 = gen.flow(X1,X2, batch_size=batch_size,seed=666)     while True:         X1i = genX1.next()         X2i = genX2.next()         #Assert arrasy are equal - this was for peace of mind, but slows down training         #np.testing.assert_array_equal(X1i[0],X2i[0])         yield [X1i[0], X2i[1]], X1i[1] 

Answer 2

I have an implementation for multiple inputs for TimeseriesGenerator that I have adapted it (I have not been able to test it unfortunately) to meet this example with ImageDataGenerator. My approach was to build a wrapper class for the multiple generators from keras.utils.Sequence and then implement the base methods of it: __len__ and __getitem__:

from keras.preprocessing.image import ImageDataGenerator from keras.utils import Sequence   class MultipleInputGenerator(Sequence):     """Wrapper of 2 ImageDataGenerator"""      def __init__(self, X1, X2, Y, batch_size):         # Keras generator         self.generator = ImageDataGenerator(rotation_range=15,                                              width_shift_range=0.2,                                             height_shift_range=0.2,                                             shear_range=0.2,                                             zoom_range=0.2,                                             horizontal_flip=True,                                              fill_mode='nearest')          # Real time multiple input data augmentation         self.genX1 = self.generator.flow(X1, Y, batch_size=batch_size)         self.genX2 = self.generator.flow(X2, Y, batch_size=batch_size)      def __len__(self):         """It is mandatory to implement it on Keras Sequence"""         return self.genX1.__len__()      def __getitem__(self, index):         """Getting items from the 2 generators and packing them"""         X1_batch, Y_batch = self.genX1.__getitem__(index)         X2_batch, Y_batch = self.genX2.__getitem__(index)          X_batch = [X1_batch, X2_batch]          return X_batch, Y_batch 

You can use this generator with model.fit_generator() once the generator has been instanced.