how to process input with AVAudioEngine manual rendering mode?

Question

I'm having trouble getting AVAudioEngine manual rendering working when processing input. It's easy to get it working when there's no input node and audio comes from a player node.

Here's the code I've got, which can be pasted into a test:

- (void)testManualRendering {

    auto engine = [[AVAudioEngine alloc] init];

    auto format = [[AVAudioFormat alloc] initStandardFormatWithSampleRate:44100.0
                                                                 channels:2];

    auto inputBuffer = [[AVAudioPCMBuffer alloc] initWithPCMFormat:format
                                                     frameCapacity:1024];

    auto abl = inputBuffer.mutableAudioBufferList;
    XCTAssert(abl != NULL);
    XCTAssertEqual(abl->mNumberBuffers, 2);

    [engine connect:engine.inputNode to:engine.mainMixerNode format:format];

    NSError* error;
    [engine enableManualRenderingMode:AVAudioEngineManualRenderingModeOffline
                               format:format
                    maximumFrameCount:1024 error:&error];
    XCTAssertNil(error);
    XCTAssert([format isEqual:engine.manualRenderingFormat]);
    NSLog(@"manualRenderingFormat: %@", engine.manualRenderingFormat);

    auto success = [engine.inputNode setManualRenderingInputPCMFormat:format
                                                           inputBlock:^const AudioBufferList * _Nullable(AVAudioFrameCount inNumberOfFrames) {
        XCTAssert(abl->mBuffers[0].mDataByteSize <= inNumberOfFrames * sizeof(float));
        XCTAssert(abl->mBuffers[1].mDataByteSize <= inNumberOfFrames * sizeof(float));
        return abl;
    }];
    XCTAssert(success);

    [engine startAndReturnError:&error];
    XCTAssertNil(error);

    auto outputBuffer = [[AVAudioPCMBuffer alloc] initWithPCMFormat:format
                                                      frameCapacity:1024];
    XCTAssertNotNil(outputBuffer);

    XCTAssert(engine.isInManualRenderingMode);
    auto status = [engine renderOffline:32 toBuffer:outputBuffer error:&error];
    if(status == AVAudioEngineManualRenderingStatusInsufficientDataFromInputNode) {
        printf("manual rendering failed: AVAudioEngineManualRenderingStatusInsufficientDataFromInputNode\n");
    }
    XCTAssertEqual(status, AVAudioEngineManualRenderingStatusSuccess);

    if (error) {
        NSLog(@"error: %@", error);
    }
    XCTAssertNil(error);
}

I'm getting AVAudioEngineManualRenderingStatusInsufficientDataFromInputNode but the input buffer has plenty of data. What am I missing? Having trouble finding example code for this.

Answer 1

I've mulled over this for a few afternoons, and after some playing around, I think I have a solution for your problem and some answers as to why this isn't working as you're expecting.

AVAudioEngine's inputNode defaults to AVAudioInputNode or the input of the system's audio interface. When the engine is in manual rendering mode, both the input and output nodes are disconnected from the audio interface, but a lot of the settings stay in place. So necessarily hwFormat.sampleRate must == format.sampleRate, otherwise the [engine connect] call will fail.

Because the audio interface is disconnected, we are not going to get data from the input, so we can use setManualRenderingInput to determine how we are supplying data. We can ask [engine renderOffline] to render a frameCapacity, apparently however, this is not guaranteed to be == inNumberOfFrames value passed within our inputBlock it changes apparently based on the selected audio device and it's respective sampleRate. For the renderOffline call to be successful AudioBuffer.mDataByteSize MUST == inNumberOfFrames * sizeof(float) otherwise the renderOffline call will throw insufficient data. As we cannot ensure the value of inNumberOfFrames before the call to renderOffline, the allocation of the AudioBuffers must wait until the inputBlock.

I'm sure there is some stuff I've missed... but anyway... try the changes below and see if this works for you.

- (void)testManualRendering {

    auto engine = [[AVAudioEngine alloc] init];
    auto hwFormat = [engine.inputNode inputFormatForBus:0];
    NSLog(@"HW Format: %@", hwFormat);
        
    ///Note: SampleRate will have to match the systems audio interface input sample rate or engine connect will throw an error
    auto format = [[AVAudioFormat alloc] initStandardFormatWithSampleRate:[hwFormat sampleRate] channels:2];
 
    NSLog(@"Input Node: %@", [engine inputNode]);
    
    auto frameCapacity = 1024;
    
    AudioBufferList *abl = (AudioBufferList *)calloc(sizeof(AudioBufferList) + sizeof(AudioBuffer), 1);
    XCTAssert(NULL != abl, @"Unable to allocate memory");
    abl->mNumberBuffers = [format channelCount];
    

    XCTAssertEqual(abl->mNumberBuffers, [format channelCount]);
    
    [engine connect:engine.inputNode to:engine.mainMixerNode format:format];

    NSError* error;
    [engine enableManualRenderingMode:AVAudioEngineManualRenderingModeOffline
                               format:format
                    maximumFrameCount:frameCapacity error:&error];
    XCTAssertNil(error);
    XCTAssert([format isEqual:engine.manualRenderingFormat]);
    NSLog(@"manualRenderingFormat: %@", engine.manualRenderingFormat);

    auto success = [engine.inputNode setManualRenderingInputPCMFormat:format
                                                           inputBlock:^const AudioBufferList * _Nullable(AVAudioFrameCount inNumberOfFrames) {
        
        auto byteSize = UInt32((inNumberOfFrames) * sizeof(float));
        
        unsigned i, j;
        
        for(i=0; i<abl->mNumberBuffers; ++i){
            abl->mBuffers[i].mData = calloc(sizeof(float), inNumberOfFrames);
            XCTAssert(NULL != abl->mBuffers[i].mData, @"Unable to allocate memory.");
            abl->mBuffers[i].mDataByteSize = byteSize;
            abl->mBuffers[i].mNumberChannels = 1;
            
            XCTAssert( abl->mBuffers[i].mDataByteSize > 0);
            XCTAssert( abl->mBuffers[i].mDataByteSize >= inNumberOfFrames * sizeof(float));

            float *ptr = (float *)abl->mBuffers[i].mData;
            
            //possibly load in some random data? WHY NOT!
            for(j=0; j<inNumberOfFrames; ++j){
                *ptr = (rand() / float(RAND_MAX)) * 0.8;
                ptr++;
            }
            
        }
        
        NSLog(@"Number of input frames: %i, size of frames (bytes): %lu, buffer capacity: %i", inNumberOfFrames, inNumberOfFrames * sizeof(float), abl->mBuffers[0].mDataByteSize);
        
        return abl;
    }];
    XCTAssert(success);

    [engine startAndReturnError:&error];
    XCTAssertNil(error);

    auto outputBuffer = [[AVAudioPCMBuffer alloc] initWithPCMFormat:format
                                                      frameCapacity:frameCapacity];
    XCTAssertNotNil(outputBuffer);

    XCTAssert(engine.isInManualRenderingMode);
    auto status = [engine renderOffline:frameCapacity toBuffer:outputBuffer error:&error];
    
    if(status == AVAudioEngineManualRenderingStatusInsufficientDataFromInputNode) {
        printf("manual rendering failed: AVAudioEngineManualRenderingStatusInsufficientDataFromInputNode\n");
    }
    XCTAssertEqual(status, AVAudioEngineManualRenderingStatusSuccess);

    if (error) {
        NSLog(@"error: %@", error);
    }
    XCTAssertNil(error);
    
    //free buffers
    unsigned i;
    for(i=0; i<abl->mNumberBuffers; ++i){
        free(abl->mBuffers[i].mData);
    }
    free(abl);
}