Keras training is slow when using ImageDataGenerator

Question

So I just got started using Keras, I first tried the MNIST dataset, and everything was quick 5 epoch in only few seconds (60k images).

Now I'm trying to train a CNN on a car dataset that has around 8000 image, I used ImageDataGenerator to resize the image into the same size (128 * 128) and to get it from a dataframe that has the filename and the class name for each image, here is the part responsible for this:

datagen=ImageDataGenerator(validation_split=0.2)
train_generator = datagen.flow_from_dataframe(dataframe=df_train, directory="cars/cars_train/cars_train/", x_col="fname", y_col="class_name", class_mode="categorical", target_size=(128,128), batch_size=32,subset="training")
test_generator = datagen.flow_from_dataframe(dataframe=df_train, directory="cars/cars_train/cars_train/", x_col="fname", y_col="class_name", class_mode="categorical", target_size=(128,128), batch_size=32,subset="validation")

As you can see I'm not doing anything intensive except resizing the dataset. When I use model.fit on the ImageDataGenerator, the first epoch take around 2800 seconds to finish and the other epoch take around 60 - 70 second only (1 minute). The NN is not complicated, it's only 2 convolutional layers + one dense layer.

nb_epochs = 4
bs = 32
model.fit(
    train_generator,
    steps_per_epoch = train_generator.samples//bs,
    validation_data = test_generator, 
    validation_steps = test_generator.samples//bs,
    epochs = nb_epochs)

I would like to know if it's normal that using ImageDataGenerator, training the network is slow especially on the first epoch ? If so is there a faster more efficient way to resize images and preprocess them before training the network ?

Btw, I'm using Google Colab, and I enabled the GPU and checked that tensorflow is using the GPU

Here's the link for the notebook, it's quite messy, I'm sorry: https://colab.research.google.com/drive/1FqPeqIhKo_lUjhoB2jlne1HnwItOJkIW?usp=sharing

Answer 1

if you can specify workers=8 as kwarg in fit, try that.

if not, swap fit with the deprecated method fit_generator and see if you get a performance increase if you up the workers. https://www.tensorflow.org/api_docs/python/tf/keras/Model

model.fit_generator(
    train_generator,
    workers=8,
    steps_per_epoch = train_generator.samples//bs,
    validation_data = test_generator, 
    validation_steps = test_generator.samples//bs,
    epochs = nb_epochs)

Alternatively, you could add an average or max pool at the top of your neural net so down-sampling is built in the neural net, thereby leveraging gpu to perform the down-sampling.