iOS Swift playing audio (aac) from network stream

Question

I'm developing an iOS application and I'm quite new to iOS development. So far I have implemented a h264 decoder from network stream using VideoToolbox, which was quite hard. Now I need to play an audio stream that comes from network, but with no file involved, just a raw AAC stream read directly from the socket. This streams comes from the output of a ffmpeg instance. The problem is that I don't know how to start with this, it seems there is little information about this topic. I have already tried with AVAudioPlayer but found just silence. I think I have first need to decompress the packets from the stream, just like with the h264 decoder.

I have been trying also with AVAudioEngine and AVAudioPlayerNode but no sucess, same as with AVAudioPlayer. Can someone provide me some guidance? Maybe AudioToolbox? AudioQueue?

Thank you very much for the help :)

Edit: I'm playing around with AVAudioCompressedBuffer and having no error using AVAudioEngine and AVAudioNode. But, I don't know what this output means:

inBuffer: <AVAudioCompressedBuffer@0x6040004039f0: 0/1024 bytes>

Does this mean that the buffer is empty? I have been trying to feed this buffer in several ways, but always returns something like 0/1024. I think I'm not doing this right:

compressedBuffer.mutableAudioBufferList.pointee = audioBufferList

Any idea?

Thank you!

Edit 2: I'm editing for reflecting my code for decompressing the buffer. Maybe some one can point me in the right direction. Note: The packet that is ingested by this function actually is passed without the ADTS header (9 bytes) but I have also tried passing it with the header.

func decodeCompressedPacket(packet: Data) -> AVAudioPCMBuffer {

    var packetCopy = packet
    var streamDescription: AudioStreamBasicDescription = AudioStreamBasicDescription.init(mSampleRate: 44100, mFormatID: kAudioFormatMPEG4AAC, mFormatFlags: UInt32(MPEG4ObjectID.AAC_LC.rawValue), mBytesPerPacket: 0, mFramesPerPacket: 1024, mBytesPerFrame: 0, mChannelsPerFrame: 1, mBitsPerChannel: 0, mReserved: 0)
    let audioFormat = AVAudioFormat.init(streamDescription: &streamDescription)
    let compressedBuffer = AVAudioCompressedBuffer.init(format: audioFormat!, packetCapacity: 1, maximumPacketSize: 1024)

    print("packetCopy count: \(packetCopy.count)")
    var audioBuffer: AudioBuffer = AudioBuffer.init(mNumberChannels: 1, mDataByteSize: UInt32(packetCopy.count), mData: &packetCopy)
    var audioBufferList: AudioBufferList = AudioBufferList.init(mNumberBuffers: 1, mBuffers: audioBuffer)
    var mNumberBuffers = 1
    var packetSize = packetCopy.count
    // memcpy(&compressedBuffer.mutableAudioBufferList[0].mBuffers, &audioBuffer, MemoryLayout<AudioBuffer>.size)
    // memcpy(&compressedBuffer.mutableAudioBufferList[0].mBuffers.mDataByteSize, &packetSize, MemoryLayout<Int>.size)
    // memcpy(&compressedBuffer.mutableAudioBufferList[0].mNumberBuffers, &mNumberBuffers, MemoryLayout<UInt32>.size)

    // compressedBuffer.mutableAudioBufferList.pointee = audioBufferList

    var bufferPointer = compressedBuffer.data

    for byte in packetCopy {
        memset(compressedBuffer.mutableAudioBufferList[0].mBuffers.mData, Int32(byte), MemoryLayout<UInt8>.size)
    }

    print("mBuffers: \(compressedBuffer.audioBufferList[0].mBuffers.mNumberChannels)")
    print("mBuffers: \(compressedBuffer.audioBufferList[0].mBuffers.mDataByteSize)")
    print("mBuffers: \(compressedBuffer.audioBufferList[0].mBuffers.mData)")


    var uncompressedBuffer = uncompress(inBuffer: compressedBuffer)
    print("uncompressedBuffer: \(uncompressedBuffer)")
    return uncompressedBuffer
}

Answer 1

So you are right in thinking you will (most likely) need to decompress the packets received from the stream. The idea is to get them to raw PCM format so that this can be sent directly to the audio output. This way you could also apply any DSP / audio manipulation you could want to the audio stream.

---

As you mentioned, you will probably need to be looking into the AudioQueue direction and the Apple Docs provide a good example of streaming audio in realtime, although this is in obj-c (in this case I think it may be a good idea to carry this out in obj-c). This is probably the best place to get started (interfacing the obj-c to swift is super simple).

---

Looking again at it in Swift there is the class AVAudioCompressedBuffer which seems to handle AAC for your case (would not need to decode the AAC if you get this to work), however there is no direct method for setting the buffer as it is intended for just being a storage container, I believe. Here's a working example of someone using the AVAudioCompressedBuffer along with an AVAudioFile (maybe you could buffer everything into files in background threads? I think it would be too much IO overhead).

However, if you tackle this in obj-c there is a post on how to set the AVAudioPCMBuffer (maybe works with AVAudioCompressedBuffer?) directly through memset (kind of digusting but at the same time lovely as an embedded programmer myself).

// make a silent stereo buffer  
AVAudioChannelLayout *chLayout = [[AVAudioChannelLayout alloc] initWithLayoutTag:kAudioChannelLayoutTag_Stereo];  
AVAudioFormat *chFormat = [[AVAudioFormat alloc] initWithCommonFormat:AVAudioPCMFormatFloat32  
                                                          sampleRate:44100.0  
                                                          interleaved:NO  
                                                        channelLayout:chLayout];  

AVAudioPCMBuffer *thePCMBuffer = [[AVAudioPCMBuffer alloc] initWithPCMFormat:chFormat frameCapacity:1024];  

thePCMBuffer.frameLength = thePCMBuffer.frameCapacity;  

for (AVAudioChannelCount ch = 0; ch < chFormat.channelCount; ++ch) {  
    memset(thePCMBuffer.floatChannelData[ch], 0, thePCMBuffer.frameLength * chFormat.streamDescription->mBytesPerFrame);  
}  

---

I know this is a lot to take and no way seems like a simple solution, but I think the obj-c AudioQueue technique would be my first stop!

Hope this helps!