Keras ImageDataGenerator Fit causes memory leak

Question

I'm on Keras 2.2.2 and I'm trying to generate augmentations of my training data with zca_whitening and an ImageDataGenerator. But when I try to fit the generator (which is mandatory when using zca_whitening) the python process eats more and more memory (100Gb+) until it gets killed by the system.

This small example can cause the leak:

import numpy as np
from keras.preprocessing.image import ImageDataGenerator

def cause_leak():
    idg = ImageDataGenerator(zca_whitening = True)
    random_sample = np.random.random((1, 250, 250, 3))
    idg.fit(random_sample)

cause_leak()

Update: Yesterday this was marked as a bug in the Keras repository.

Answer 1

As discussed in this issue this is not a memory leak as computing Singular Value Decomposition on the matrix with (250 * 250 * 3) i.e., 187000 elements is memory intensive. Unfortunately, there is no immediate work around for this issue as of now as problem lies with the calculation of sigma matrix as mentioned here