Memory Leak from iterating Opencv frames

Question

I am using the java wrapper of OpenCV. I tried to write an Iterator over frames of a film. My problem is that the iterator is a huge memory leak. Here is a very simplified version of the iterator, which has this leak:

public static final class SimpleIt implements Iterator<Mat> {

    private final VideoCapture capture;
    boolean hasNext;

    public SimpleIt(final VideoCapture capture) {
        this.capture = capture;
        hasNext = capture.grab();
    }

    @Override
    public boolean hasNext() {
        return hasNext;
    }

    @Override
    public Mat next() {
        final Mat mat = new Mat();
        capture.retrieve(mat);
        hasNext = capture.grab();
        return mat;
    }
}

I Iterate over this code using this loop:

    final VideoCapture vc = new VideoCapture("/path/to/file");
    final SimpleIt it = new SimpleIt(vc);
    while (it.hasNext) {
        it.next();
    }

Just iterating will increase memory consumption linear. I see that the problem is the first line in the next()-Method. It always creates a new Mat. But speaking of java alone, this Mat will run out of scope as soon as the iterating code iterates to the next image.

I could overcome the problem, by not using a new Mat every time, but overwriting always the same Mat-Object, like this:

    private final VideoCapture capture;
    private final Mat mat = new Mat();
    boolean hasNext;

    @Override
    public Mat next() {
        capture.retrieve(mat);
        hasNext = capture.grab();
        return mat;
    }

But now the last frame which was given by the iterator will be overwritten. Thus, I cannot hold it outside for later use, if I am interested in this single frame. I could copy it, of course, but that would also be expensive.

I assume that the problem is that the garbage collector will not destroy the Mat objects, because it does not recognize the memory consumption, since it is not java heap space. Calling mat.release() in the loop will help, but of course in real code this means I will have no garbage collection for my Mat objects.

Anybody has an idea how to do it?

Edit:

Since it seems not to be clear what the problem with my second solution is, I write it down more explicitly. Think about the following code, using the iterator:

    final VideoCapture vc = new VideoCapture("/path/to/file");
    final SimpleIt it = new SimpleIt(vc);
    int i = 0;
    Mat save = null;
    while (it.hasNext) {
        final Mat next = it.next();
        if (i == 10) {
            save = next;
            Highgui.imwrite("/path/to/10.png", save);
        } else if (i == 30) {
            Highgui.imwrite("/path/to/30.png", save);
        }
        i++;
    }

With the second version of the iterator, 10.png, and 30.png will be different images. But that's obviously not what was intended.

Answer 1

You should really call mat.release().

I have very similar problem to yours in my application. The frame rate was so high that java heap grew up to total available system memory, which sometimes led to JVM crash. GC was simply too slow, and I didn't have any mechanism to check of available memory and wait if that wasn't sufficient.

One solution was to decrease the frame rate by simply using Thread.sleep() which of course didn't seem to be acceptable. But it helped GC to do it's job on time.

Finally using mat.release() fixed the problem.

You don't have to worry about garbage collection of Mat objects, because this call deallocates only underlying data. Java object wrapper will be disposed by GC when the right time comes.

Answer 2

I just want to add my $0.02 as I bumped into this issue when writing an application that will run for a long.

Mat.release() is called automatically when the Java Mat-wrapper is garbage collected. However, since the Java wrapper is very small compared to the natively allocated object, it might not be garbage-collected fast enough.

Therefore, you could either do Mat.release() when you know you are done with an object or call System.gc() at regular intervals to force removal of unused objects.