dlib train_object_detector immense amounts of RAM usage

Question

I am using dlib's train_object_detector for face detection and I have roughly 6k images in a folder with which I am trying to train my model.

Also, I am using dlib's example python code(train_object_detector.py) for this purpose.

But the thing is, the program's RAM usage is insane. For roughly 300 images, it required approximately 15GB RAM and right now with my 6k images, I'm stuck.

For 6k images, while training, it required more than 100GBs of RAM and eventually program killed itself.

Has it always been like this? or am I doing something wrong? Is it normal to have this much of RAM usage?

It is almost not modified at all and pretty much same with the example code from dlib.

Note: The sizes of the images are between 10-100 KB.

Here is the code I'm using (remote): http://pastebin.com/WipU8qgq Here's the code:

import os
import sys
import glob
import dlib
from skimage import io


if len(sys.argv) != 4:
        print(
        "Give the path to the faces directory as the argument to this "
        "program with training and test xml files in order. For example: \n"
        "    ./train_object_detector_modified.py ../faces ../faces/training.xml ../faces/testing.xml")
    exit()
faces_folder = sys.argv[1]
training_xml_path = sys.argv[2]
testing_xml_path = sys.argv[3]

options = dlib.simple_object_detector_training_options()
options.add_left_right_image_flips = True
options.C = 5
options.num_threads = 8
options.be_verbose = True

dlib.train_simple_object_detector(training_xml_path, "detector.svm", options)
print 'training end'

print("")  # Print blank line to create gap from previous output
print("Training accuracy: {}".format(
    dlib.test_simple_object_detector(training_xml_path, "detector.svm")))

print("Testing accuracy: {}".format(
    dlib.test_simple_object_detector(testing_xml_path, "detector.svm")))


'''
# Now let's use the detector as you would in a normal application.  First we
# will load it from disk.
detector = dlib.simple_object_detector("detector.svm")

# We can look at the HOG filter we learned.  It should look like a face.  Neat!
win_det = dlib.image_window()
win_det.set_image(detector)

# Now let's run the detector over the images in the faces folder and display the
# results.
print("Showing detections on the images in the faces folder...")
win = dlib.image_window()
for f in glob.glob(os.path.join(faces_folder, "*.jpg")):
   print("Processing file: {}".format(f))
   img = io.imread(f)
   dets = detector(img)
   print("Number of faces detected: {}".format(len(dets)))
   for k, d in enumerate(dets):
       print("Detection {}: Left: {} Top: {} Right: {} Bottom: {}".format(
           k, d.left(), d.top(), d.right(), d.bottom()))

   win.clear_overlay()
   win.set_image(img)
   win.add_overlay(dets)
   dlib.hit_enter_to_continue()
'''

Answer 1

This is happening because you have a combination of big images and/or small bounding boxes. By default, dlib.train_simple_object_detector uses a detection window that is 6400 pixels in size. If images contain target boxes much smaller than this then those images are upsampled to make the objects big enough.

All these settings are fields in the options object.