Tensorflow Allocation Memory: Allocation of 38535168 exceeds 10% of system memory

Question

Using ResNet50 pre-trained Weights I am trying to build a classifier. The code base is fully implemented in Keras high-level Tensorflow API. The complete code is posted in the below GitHub Link.

Source Code: Classification Using RestNet50 Architecture

The file size of the pre-trained model is 94.7mb.

I loaded the pre-trained file

new_model = Sequential()  new_model.add(ResNet50(include_top=False,                 pooling='avg',                 weights=resnet_weight_paths)) 

and fit the model

train_generator = data_generator.flow_from_directory(     'path_to_the_training_set',     target_size = (IMG_SIZE,IMG_SIZE),     batch_size = 12,     class_mode = 'categorical'     )  validation_generator = data_generator.flow_from_directory(     'path_to_the_validation_set',     target_size = (IMG_SIZE,IMG_SIZE),     class_mode = 'categorical'     )  #compile the model  new_model.fit_generator(     train_generator,     steps_per_epoch = 3,     validation_data = validation_generator,     validation_steps = 1 ) 

and in the Training dataset, I have two folders dog and cat, each holder almost 10,000 images. When I compiled the script, I get the following error

Epoch 1/1 2018-05-12 13:04:45.847298: W tensorflow/core/framework/allocator.cc:101] Allocation of 38535168 exceeds 10% of system memory. 2018-05-12 13:04:46.845021: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:47.552176: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:48.199240: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:48.918930: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:49.274137: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:49.647061: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:50.028839: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:50.413735: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory.

Any ideas to optimize the way to load the pre-trained model (or) get rid of this warning message?

Thanks!

Answer 1

Try reducing batch_size attribute to a small number(like 1,2 or 3). Example:

train_generator = data_generator.flow_from_directory(     'path_to_the_training_set',     target_size = (IMG_SIZE,IMG_SIZE),     batch_size = 2,     class_mode = 'categorical'     )