Load data on demand using flow_from_directory for AutoEncoder

Question

I am trying to train an AutoEncoder on some image data. The dataset is so huge so that it won't fit in memory. So obviously I want to load the data from the directory on demand with the help of flow_from_directory in Keras.

My dataset is in the following structure

./Dataset/
    ./Train/
        ../1.jpg
        ../2.jpg

and so on.

I tried to use flow_from_directory like this

train_generator = datagen.flow_from_directory(
    TRAIN_FOLDER,
    target_size = (256, 256),
    color_mode = 'rgb',
    batch_size = batch_size,
    class_mode = 'input')

This gives me an output Found 0 images belonging to 0 classes. I will get a ZeroDivisionError if I try to fit the model with this generator.

I have used the flow_from_directory, flow and flow_from_dataframe in various occasions but in those cases, I was going for a classification problem and had n folders in the directory for n classes.

How can I load images from the directory on demand for training auto encoder.? From the Keras docs over here I saw

class_mode : "input" will be images identical to input images (mainly used to work with autoencoders)

But this is not fixing the problem either.

One workaround I found was making another folder inside the train and moving all the files into that. Is there any direct method other than this.?

Answer 1

The Keras docs are indeed not very precise here and the way it actually works is not intuitive (at least to me) ...

Here, even though you are using class_mode='input' which means that there are no classes (or each picture is its own class, however you like to phrase it), your images still have to be inside of subfolders.

So, inside of your Train folder just create another subfolder and move all images inside. Then the output will be Found xxx images belonging to 1 classes.

You can even have multiple subfolders, the generated X and Y data will be the same, just the console output will be different (and misleading): Found xxx images belonging to yy classes. This may be useful if there actually are classes and you have another model using them (using class_mode='categorical' or class_mode='binary'). Then this model can load its data from the same folder.