Accurate (Music-grade) Timing of Audio Playback [closed]

Question

I'm attempting to write an app that can arrange timed playback of some (mostly tiny) audio files. The catch: it needs to be timed with accuracy worthy of music. The sounds won't be played frequently (it's not, for example, a sampler or a drum kit), but they will need to be absolutely precisely placed. There are two parts to this question:

1. How should I organize the timing? I've heard NSTimer is inaccurate, or at least unreliable? Should I use MIDI somehow? Or a tight loop and some time-getting function with high resolution?

2. Once I've got the timing set up, and I know when to play my sounds... how should I play them? Is NSSound fast and reliable enough to maintain the accuracy needed?

Answer 1

General Info On Music Quality Audio

This is something that needs to be shouted from the mountaintops to anybody making an instrument for iOS:

If you are using MIDI, you will wrestle much less with these issues (the sound renderer had to deal with much of this), but all instruments have to deal with these requirements:

What Musicians Expect

The first thing is to make the buffers used by AudioQueue as small as possible. The default on iPad is 1024 samples, which is a painfully long 23 milliseconds. When you want to play a sound, with that buffer size, it will take from 0 to 23 ms for a response, even if the computer is infinitely fast. If the buffer is more than 256 samples long, you won't be able to do much of a real instrument with it. In practice, also add as much as 30ms on top of that for overhead (from touch response to graphics, etc).

//   256 / ( 44.1khz) ~  5.8ms 
float latency = 0.005; //in seconds gives a 256 sample buffer
//on an infinitely fast computer, we'd make sure we got a buffer of size 1.
// but it will take too much CPU. :-)  some latency is inevitable.
OSStatus status = AudioSessionSetProperty(
  kAudioSessionProperty_PreferredHardwareIOBufferDuration,
  sizeof(latency),&latency
);

• Latency - Pro musicians expect finger-to-ear latency to be about 8 milliseconds. That seems like a fantasy on the iPad for me, but that's the goal. Just for playing fast, think of 120 (quarternote) beats per minute and target 4x accuracy of 16th notes, etc. Even when not playing fast, musicians are irritated by high latency, and will stop using an otherwise great instrument that goes above 100ms(!!!!) Anything above 30ms, they are unhappily going into pro-tools and time-shifting the recorded sound a few ms back to fix your screw-up. The better apps on the store get to about 30ms. Some really good ones are still OK at about 50ms.

  Measure latency by placing a microphone on top of the surface of the ipad, and running it into the left channel of a mixer. Then run the ipad out into the right channel. Record the sound in stereo. Then bring it up into a sound editor that lets you view stereo waveforms. You can look at the time that the finger hit the touchscreen, then to the time that the electronic sound began. That distance is latency. The variance in it is jitter.

• Jitter - Jitter is variance in latency. Jitter of more than a few milliseconds makes rhythm sound wrong. I got to about 5ms. It's very directly tied to the AudioQueue buffer size. Example with 1024 sample buffer: If you press your finger down at some random point in time that the buffer then the expectation is to be at sample 512 when you need to respond. So it varies from 0 to 1024 ms minimum, but it's 512 on average... 12ms of waiting for the buffer to finish as an average case, before you add in latency from other sources.

• Touch Latency - It's most important that touchesBegan get serviced as quickly as possible. Even if that means that touchesMoved, touchesEnded, touchesCancelled take a little longer. So, you might end up having some asymmetry in the workload to achieve this. This was a confusing revelation to me at first, because the profiler was telling me that there wasn't much time being spent in touchesBegan, yet it can make a big difference to defer some work until the touch ended.

• OpenGL - You should at least consider it. I am not convinced that it is as easy to hit these latency goals when using UIKit directly, and used OpenGL on my main app.

• Real-Time Clock - For things that aren't really instruments, but have stringent beat-sync requirements like drum machines, you could keep track of what time it really is and make the samples track to the clock and assume that there is drift when you aren't watching the clock. But I got tired of fighting that battle on my older MIDI instruments ... if you have MIDI, then MIDI does clock signals; it might be the only thing that works to sync a couple of electronic instruments together.

• Sound Skips - Everything that your app does competes for CPU time. Given that you have made your sound engine as efficient as you can, making the buffers smaller uses more CPU. You might have to trade a little latency for better continuity. I have my bias, cut out eye-candy nonsense and live within your means if the sound engine will get more cycles back.

• Timers. Using CADisplayLink. I haven't had to think about it much. This is what I have.

- (NSInteger)animationFrameInterval
{
    return animationFrameInterval;
}

- (void)setAnimationFrameInterval:(NSInteger)frameInterval
{
    if (frameInterval >= 16)
    {
        animationFrameInterval = frameInterval;

        if (animating)
        {
            [self stopAnimation];
            [self startAnimation];
        }
    }
}

- (void)startAnimation
{
    if (!animating)
    {
      displayLink = [NSClassFromString(@"CADisplayLink")
            displayLinkWithTarget:self
            selector:@selector(drawView:)];
        [displayLink setFrameInterval:animationFrameInterval];
        [displayLink addToRunLoop:[NSRunLoop
              currentRunLoop]
            forMode:NSDefaultRunLoopMode];

        animating = TRUE;
    }
}

- (void)stopAnimation
{
    if (animating)
    {
      [displayLink invalidate];
      displayLink = nil;

        animating = FALSE;
    }
}

I am not making this stuff up. Please don't put up "music instruments" in the store until you are addressing these issues. Jordan Rudess, Leon Gruenbaum, Chris Dudley and other pro musicians used to bring these issues up constantly when talking about iPad/iPhone apps. The situation seems to have improved somewhat, but I don't think most developers know what numbers real musicians expect.