appendSampleBuffer with an audio AVAssetWriterInput "leaks" memory until endSessionAtSourceTime

Question

I have a strange memory "leak" with AVAssetWriterInput appendSampleBuffer. I'm writing video and audio at the same time, so I have one AVAssetWriter with two inputs, one for video and one for audio:

self.videoWriter = [[[AVAssetWriter alloc] initWithURL:[self.currentVideo currentVideoClipLocalURL]
                                              fileType:AVFileTypeMPEG4
                                                 error:&error] autorelease];
...
self.videoWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
                                                           outputSettings:videoSettings];
self.videoWriterInput.expectsMediaDataInRealTime = YES;
[self.videoWriter addInput:self.videoWriterInput];
...
self.audioWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio
                                                           outputSettings:audioSettings];
self.audioWriterInput.expectsMediaDataInRealTime = YES;
[self.videoWriter addInput:self.audioWriterInput];

I start writing and everything works fine on the surface. The video and audio get written and are aligned, etc. However, I put my code through the Allocations Instrument and noticed the following:

The audio bytes are getting retained in memory, as I'll prove in a second. That's the ramp up in memory. The audio bytes are only released after I call [self.videoWriter endSessionAtSourceTime:...], which you see as the dramatic drop in memory usage. Here is my audio writing code, which is dispatched as a block onto a serial queue:

@autoreleasepool
{
    // The objects that will hold the audio data
    CMSampleBufferRef sampleBuffer;
    CMBlockBufferRef  blockBuffer1;
    CMBlockBufferRef  blockBuffer2;

    size_t nbytes = numSamples * asbd_.mBytesPerPacket;

    OSStatus status = noErr;
    status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault,
                                                data,
                                                nbytes,
                                                kCFAllocatorNull,
                                                NULL,
                                                0,
                                                nbytes,
                                                kCMBlockBufferAssureMemoryNowFlag,
                                                &blockBuffer1);

    if (status != noErr)
    {
        NLog(@"CMBlockBufferCreateWithMemoryBlock error at buffer 1");
        return;
    }

    status = CMBlockBufferCreateContiguous(kCFAllocatorDefault,
                                           blockBuffer1,
                                           kCFAllocatorDefault,
                                           NULL,
                                           0,
                                           nbytes,
                                           kCMBlockBufferAssureMemoryNowFlag | kCMBlockBufferAlwaysCopyDataFlag,
                                           &blockBuffer2);

    if (status != noErr)
    {
        NSLog(@"CMBlockBufferCreateWithMemoryBlock error at buffer 2");
        CFRelease(blockBuffer1);
        return;
    }

    // Finally, create the CMSampleBufferRef
    status = CMAudioSampleBufferCreateWithPacketDescriptions(kCFAllocatorDefault,
                                                             blockBuffer2,
                                                             YES,   // Yes data is ready
                                                             NULL,  // No callback needed to make data ready
                                                             NULL,
                                                             audioFormatDescription_,
                                                             1,
                                                             timestamp,
                                                             NULL,
                                                             &sampleBuffer);


    if (status != noErr)
    {
        NSLog(@"CMAudioSampleBufferCreateWithPacketDescriptions error.");
        CFRelease(blockBuffer1);
        CFRelease(blockBuffer2);
        return;
    }

    if ([self.audioWriterInput isReadyForMoreMediaData])
    {
        if (![self.audioWriterInput appendSampleBuffer:sampleBuffer])
        {
            NSLog(@"Couldn't append audio sample buffer: %d", numAudioCallbacks_);
        }
    } else {
        NSLog(@"AudioWriterInput isn't ready for more data.");
    }

    // One release per create
    CFRelease(blockBuffer1);
    CFRelease(blockBuffer2);
    CFRelease(sampleBuffer);
}

As you can see, I'm releasing each buffer once per create. I've traced the "leak" down to the line where the audio buffers are appended:

[self.audioWriterInput appendSampleBuffer:sampleBuffer]

I proved this to myself by commenting out that line, after which I get the following "leak-free" Allocations graph (although the recorded video now has no audio now, of course):

I tried one other thing, which is to add back the appendSamplebuffer line and instead double-release blockBuffer2:

CFRelease(blockBuffer1);
CFRelease(blockBuffer2);
CFRelease(blockBuffer2); // Double release to test the hypothesis that appendSamplebuffer is retaining this
CFRelease(sampleBuffer);

Doing this did not cause a double-free, indicating that blockBuffer2's retain count at that point is 2. This produced the same "leak-free" allocations graph, with the exception that when I called [self.videoWriter endSessionAtSourceTime:...], I get a crash from a double-release (indicating that self.videoWriter is trying to release all of its pointers to the blockBuffer2s that have been passed in).

If instead, I try the following:

CFRelease(blockBuffer1);
CFRelease(blockBuffer2);
CMSampleBufferInvalidate(sampleBuffer); // Invalidate sample buffer
CFRelease(sampleBuffer);

then [self.audioWriterInput appendSampleBuffer:sampleBuffer] and the call to append video frames begin to fail for every call after that.

So my conclusion is that AVAssetWriter or AVAssetWriterInput is retaining blockBuffer2 until the video has finished recording. Obviously, this can cause real memory problems if the video is recording for long enough. Am I doing something wrong?

Edit: The audio bytes I'm getting are PCM format, whereas the video format I'm writing is MPEG4 and the audio format for that video is MPEG4AAC. Is it possible that the video writer is performing the PCM --> AAC format on the fly, and that's why it's getting buffered?

Answer 1

Since you have been waiting a month for an answer I'll give you a less than ideal but workable answer.

You could use the ExtendedAudioFile functions to write a separate file. Then you could just playback the video and audio together with an AVComposition. I think you might be able to use AVFoundation to composite the caf and video together without reencoding if you need them composited at the end of recording.

That will get you off and running, then you can solve the memory leak at your leisure.