How does Tensorflow's DirectoryIterator work?

Question

I am used to using something like model.fix(train_data,train_labels, epochs=10) where i use glob to read a folder full of images into RAM. I wanted to read direct from the HDD as the training happens. I found:

https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/DirectoryIterator

Only I do not know how it works exactly. I have scoured the internet for more help then the documentation linked but I haven't found any. I have the labels and directories in the DirectoryIterator. I just don't know how to feed the DirectoryIterator into my model?

The code shows what I have done so far. I also tried to use a tensorflow sess and feed the DirectoryIterator as a feed_dict. Code is messy, just been trying this and that. In the code I try to use fit_generator to fit the DirectoryIterator.

import numpy as np
import pandas as pd
import tensorflow as tf
import tensorflow.keras as keras
import cv2 as ocv
import glob
import matplotlib.pyplot as plt
from tensorflow import image
import glob

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Plot inline
%matplotlib inline

# Load an color image in 1-colour 0-grayscale -1-bw
img = ocv.imread('C:/Users/ew/Documents/Python Scripts/Noodles/my.png',1)
RGB_im = ocv.cvtColor(img, ocv.COLOR_BGR2RGB)
img.shape
plt.imshow(RGB_im)

cv_img = []
for img in glob.glob("C:\\Users\\EW\\pictures\\Noodles\\Banana\\*.jpg"):
    cv_img.append(img)
    #n= ocv.imread(img)
    #cv_img.append(n)
print(cv_img[1])

image_data_generator = keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
                                       samplewise_center=False,
                                       featurewise_std_normalization=False,
                                       samplewise_std_normalization=False,
                                       zca_whitening=False, zca_epsilon=1e-06,
                                       rotation_range=0,
                                       width_shift_range=0.0,
                                       height_shift_range=0.0,
                                       brightness_range=None,
                                       shear_range=0.0,
                                       zoom_range=0.0,
                                       channel_shift_range=0.0,
                                       fill_mode='nearest',
                                       cval=0.0,
                                       horizontal_flip=False,
                                       vertical_flip=False,
                                       rescale=None,
                                       preprocessing_function=None,
                                       data_format='channels_last',
                                       validation_split=0.3,
#                                       interpolation_order=1,
                                       dtype='float32')

noodle_data = directory = "C:\\Users\\EW\\pictures\\Noodles\\"
image_set = keras.preprocessing.image.DirectoryIterator(directory,
    image_data_generator,
    target_size=(256, 256),
    color_mode='rgb',
    classes=None,
    class_mode='categorical',
    batch_size=32,
    shuffle=True,
    seed=None,
    data_format=None,
    save_to_dir=None,
    save_prefix='',
    save_format='png',
    follow_links=False,
    subset=None,
    interpolation='nearest',
    dtype=None)

model = Sequential()

#add model layers
model.add(Dense(10, activation='relu', input_shape=(256,256)))
model.add(Dense(10, activation='relu'))
model.add(Dense(1))

model.fit_generator(noodle_data , steps_per_epoch=16, validation_data=val_it, validation_steps=8)

---> 12 model.fit_generator(prawn_data , steps_per_epoch=16, validation_data=prawn_data, validation_steps=8)
AttributeError: 'str' object has no attribute 'shape'

Answer 1

I think you should write like this:

image_data_generator = keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
                                   ...,
                                   dtype='float32')

directory = "C:\\Users\\EW\\pictures\\Noodles\\"

image_set = image_data_generator.flow_from_directory(directory,
                                                     target_size=(256, 256),
                                                     color_mode='rgb',
                                                     ...,
                                                     dtype=None)

So, you call an intance of ImageDataGenerator(), which is named image_data_generator and use its method flow_from_directory() to read from directory. And you shouldn't pass image_data_generator as your parameter in flow_from_directory.