Keras with Tensorflow: Use memory as it's needed [ResourceExhaustedError]

Question

So I'm trying to tain my CNN with mutilple datasets and it seams that when I add enough data (such as when I add multiple sets as one or when I try to add the one that has over a million samples) it throws a ResourceExhaustedError.

As for the instructions here, I tried adding

from keras.backend.tensorflow_backend import set_session
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.3
set_session(tf.Session(config=config))

to my code but this doesn't seam to make a difference. I see 0.3 after printing out config.gpu_options.per_process_gpu_memory_fraction so that part seams to be ok.

I even threw in a config.gpu_options.allow_growth = True for good mesure but it doesn't seam to want to do anything but attempt to use all the memory at once only to find that it isn't enough.

The computer I'm trying to use to train this CNN has 4 GTX1080 Ti's with 12gb of dedicated memory each.

EDIT: I'm sorry for not specifying how I was loading the data, I honestly didn't realise there was more than one way. When I was learning, they always had examples where the loaded the datasets that were already built in and it took me a while to realise how to load a self-supplied dataset.

The way I'm doing it is that I'm creating two numpy arrays . One has the path or each image and the other has the corresponding label. Here's the most basic example of this:

data_dir = "folder_name"

# There is a folder for every form and in that folder is every line of that form
for filename in glob.glob(os.path.join(data_dir, '*', '*')):

    # the format for file names are: "{author id}-{form id}-{line number}.png"
    # filename is a path to the file so .split('\\')[-1] get's the raw file name without the path and .split('-')[0] get's the author id
    authors.append(filename.split('\\')[-1].split('-')[0])
    files.append(filename)

#keras requires numpy arrays 
img_files  = np.asarray(files)
img_targets = np.asarray(authors)

Answer 1

Are you sure you're not using a giant batch_size?

"Adding data": honestly I don't know what that means and if you could please describe exactly, with code, what you're doing here, it would be of help.

The number of samples should not cause any problems with GPU memory at all. What does cause a problem is a big batch_size.

Loading a huge dataset could cause a CPU RAM problem, not related with keras/tensorflow. A problem with a numpy array that is too big. (You can test this by simply loading your data "without creating any models")

If that is your problem, you should use a generator to load batches gradually. Again, since there is absolutely no code in your question, we can't help much.

But these are two forms of simply creating a generator for images:

• Use the existing ImageDataGenerator and it's flow_from_directory() methods, explained here
• Create your own coded generator, which can be:
  • A loop with yield
  • A class derived from keras.utils.Sequence

A quick example of a loop generator:

def imageBatchGenerator(imageFiles, imageLabels, batch_size):
    while True:
        batches = len(imageFiles) // batch_size
        if len(imageFiles) % batch_size > 0:
            batches += 1

        for b in range(batches):
            start = b * batch_size
            end = (b+1) * batch_size

            images = loadTheseImagesIntoNumpy(imageFiles[start:end])
            labels = imageLabels[start:end]

            yield images,labels

Warning: even with generators, you must make sure your batch size is not too big!

Using it:

model.fit_generator(imageBatchGenerator(files,labels,batchSize), steps_per_epoch = theNumberOfBatches, epochs= ....)

Dividing your model among GPUs

You should be able to decide which layers are processed by which GPU, this "could" probably optimize your RAM usage.

Example, when creating the model:

with tf.device('/gpu:0'):
    createLayersThatGoIntoGPU0

with tf.device('/gpu:1'):
    createLayersThatGoIntoGPU1

#you will probably need to go back to a previous GPU, as you must define your layers in a proper sequence
with tf.device('/cpu:0'):
    createMoreLayersForGPU0

#and so on

I'm not sure this would be better or not, but it's worth trying too.

See more details here: https://keras.io/getting-started/faq/#how-can-i-run-a-keras-model-on-multiple-gpus