AudioUnit Input Missing Periodic Samples

Question

I have implemented an AUGraph containing a single AudioUnit for handling IO from the mic and headsets. The issue I'm having is that there are missing chunks of audio input.

I believe the samples are being lost during the hardware to software buffer exchange. I tried slowing down the sample rate of the iPhone, from 44.1 kHz to 20 kHz, to see if this would give me the missing data, but it did not produce the output I expected.

The AUGraph is setup as follows:

// Audio component description
AudioComponentDescription desc;
bzero(&desc, sizeof(AudioComponentDescription));
desc.componentType          = kAudioUnitType_Output;
desc.componentSubType       = kAudioUnitSubType_RemoteIO;
desc.componentManufacturer  = kAudioUnitManufacturer_Apple;
desc.componentFlags         = 0;
desc.componentFlagsMask     = 0;

// Stereo ASBD
AudioStreamBasicDescription stereoStreamFormat;
bzero(&stereoStreamFormat, sizeof(AudioStreamBasicDescription));
stereoStreamFormat.mSampleRate          = kSampleRate;
stereoStreamFormat.mFormatID            = kAudioFormatLinearPCM;
stereoStreamFormat.mFormatFlags         = kAudioFormatFlagsCanonical;
stereoStreamFormat.mBytesPerPacket      = 4;
stereoStreamFormat.mBytesPerFrame       = 4;
stereoStreamFormat.mFramesPerPacket     = 1;
stereoStreamFormat.mChannelsPerFrame    = 2;
stereoStreamFormat.mBitsPerChannel      = 16;

OSErr err = noErr;
@try {
    // Create new AUGraph
    err = NewAUGraph(&auGraph);
    NSAssert1(err == noErr, @"Error creating AUGraph: %hd", err);

    // Add node to AUGraph
    err = AUGraphAddNode(auGraph,
                         &desc,
                         &ioNode);
    NSAssert1(err == noErr, @"Error adding AUNode: %hd", err);

    // Open AUGraph
    err = AUGraphOpen(auGraph);
    NSAssert1(err == noErr, @"Error opening AUGraph: %hd", err);

    // Add AUGraph node info
    err = AUGraphNodeInfo(auGraph,
                          ioNode,
                          &desc,
                          &_ioUnit);
    NSAssert1(err == noErr, @"Error adding noe info to AUGraph: %hd", err);

    // Enable input, which is disabled by default.
    UInt32 enabled = 1;
    err = AudioUnitSetProperty(_ioUnit,
                         kAudioOutputUnitProperty_EnableIO,
                         kAudioUnitScope_Input,
                         kInputBus,
                         &enabled,
                         sizeof(enabled));
    NSAssert1(err == noErr, @"Error enabling input: %hd", err);

    // Apply format to input of ioUnit
    err = AudioUnitSetProperty(_ioUnit,
                         kAudioUnitProperty_StreamFormat,
                         kAudioUnitScope_Input,
                         kOutputBus,
                         &stereoStreamFormat,
                         sizeof(stereoStreamFormat));
    NSAssert1(err == noErr, @"Error setting input ASBD: %hd", err);

    // Apply format to output of ioUnit
    err = AudioUnitSetProperty(_ioUnit,
                         kAudioUnitProperty_StreamFormat,
                         kAudioUnitScope_Output,
                         kInputBus,
                         &stereoStreamFormat,
                         sizeof(stereoStreamFormat));
    NSAssert1(err == noErr, @"Error setting output ASBD: %hd", err);

    // Set hardware IO callback
    AURenderCallbackStruct callbackStruct;
    callbackStruct.inputProc = hardwareIOCallback;
    callbackStruct.inputProcRefCon = (__bridge void *)(self);
    err = AUGraphSetNodeInputCallback(auGraph,
                                      ioNode,
                                      kOutputBus,
                                      &callbackStruct);
    NSAssert1(err == noErr, @"Error setting IO callback: %hd", err);

    // Initialize AudioGraph
    err = AUGraphInitialize(auGraph);
    NSAssert1(err == noErr, @"Error initializing AUGraph: %hd", err);

    // Start audio unit
    err = AUGraphStart(auGraph);
    NSAssert1(err == noErr, @"Error starting AUGraph: %hd", err);

}
@catch (NSException *exception) {
    NSLog(@"Failed with exception: %@", exception);
}

Where kOutputBus is defined to be 0, kInputBus is 1, and kSampleRate is 44100. The IO callback function is:

IO Callback Function

static OSStatus hardwareIOCallback(void                         *inRefCon,
                               AudioUnitRenderActionFlags   *ioActionFlags,
                               const AudioTimeStamp         *inTimeStamp,
                               UInt32                       inBusNumber,
                               UInt32                       inNumberFrames,
                               AudioBufferList              *ioData) {
    // Scope reference to GSFSensorIOController class
    GSFSensorIOController *sensorIO = (__bridge GSFSensorIOController *) inRefCon;

    // Grab the samples and place them in the buffer list
    AudioUnit ioUnit = sensorIO.ioUnit;

    OSStatus result = AudioUnitRender(ioUnit,
                                      ioActionFlags,
                                      inTimeStamp,
                                      kInputBus,
                                      inNumberFrames,
                                      ioData);

    if (result != noErr) NSLog(@"Blowing it in interrupt");

    // Process input data
    [sensorIO processIO:ioData];

    // Set up power tone attributes
    float freq = 20000.00f;
    float sampleRate = kSampleRate;
    float phase = sensorIO.sinPhase;
    float sinSignal;

    double phaseInc = 2 * M_PI * freq / sampleRate;

    // Write to output buffers
    for(size_t i = 0; i < ioData->mNumberBuffers; ++i) {
        AudioBuffer buffer = ioData->mBuffers[i];
        for(size_t sampleIdx = 0; sampleIdx < inNumberFrames; ++sampleIdx) {
            // Grab sample buffer
            SInt16 *sampleBuffer = buffer.mData;

            // Generate power tone on left channel
            sinSignal = sin(phase);
            sampleBuffer[2 * sampleIdx] = (SInt16)((sinSignal * 32767.0f) /2);

            // Write to commands to micro on right channel as necessary
            if(sensorIO.newDataOut)
                sampleBuffer[2*sampleIdx + 1] = (SInt16)((sinSignal * 32767.0f) /2);
            else
                 sampleBuffer[2*sampleIdx + 1] = 0;

            phase += phaseInc;
            if (phase >= 2 * M_PI * freq) {
                 phase -= (2 * M_PI * freq);
            }
        }
    }

    // Store sine wave phase for next callback
    sensorIO.sinPhase = phase;

    return result;
}

The processIO function called within hardwareIOCallback is used to process the input and create response for the output. For debugging purposes I just have it pushing each sample of the input buffer to an NSMutableArray.

Process IO

- (void) processIO: (AudioBufferList*) bufferList {
    for (int j = 0 ; j < bufferList->mNumberBuffers ; j++) {
        AudioBuffer sourceBuffer = bufferList->mBuffers[j];
        SInt16 *buffer = (SInt16 *) bufferList->mBuffers[j].mData;

        for (int i = 0; i < (sourceBuffer.mDataByteSize / sizeof(sourceBuffer)); i++) {
            // DEBUG: Array of raw data points for printing to a file
            [self.rawInputData addObject:[NSNumber numberWithInt:buffer[i]]];
        }
    }
}

I then am writing the contents of this input buffer to a file after I have stopped the AUGraph and have all samples in the array rawInputData. I then open this file in MatLab and plot it. Here I see that the audio input is missing data (seen in the image below circled in red).

I'm out of ideas as to how to fix this issue and could really use some help understanding and fixing this problem.

Answer 1

You callback may be too slow. It's usually not recommended to use any Objective C methods (such as adding to a mutable array, or anything else that could allocate memory) inside an Audio Unit callback.