AVAudioEngine positional audio not working with .playAndRecord audio session category when using bluetooth headphones

Question

I want to be able to record audio and play back positional audio at the same time.

To do this I need to use the .playAndRecord audio session category, and simultaneous recording and playback works. However, using this category the audio file is played without being positional (i.e. it's not spatial) when using bluetooth headphones. This works as expected when using wired headphones.

If I set the audio session category to .playback, the audio played is correctly positional for both wired and bluetooth headphones, however I'm not able to simultaneously record.

I've tried various audio session categories/option but have had no luck.

import AVFoundation


class PlayerRecorder: ObservableObject {
    
    let engine = AVAudioEngine()
    let mixer = AVAudioEnvironmentNode()
    
    init() {
        let audioSession = AVAudioSession.sharedInstance()
        
        /*
         Using .playAndRecord both recording and playback works, however
         the audio that is played is NOT positional. .allowBluetooth is needed
         so that bluetooth headphones can be used.
         */
        try! audioSession.setCategory(.playAndRecord, mode: .default, options: .allowBluetooth)
        
        /*
         Using .playback the positional audio DOES work, however we are not able to record.
         */
        // try! audioSession.setCategory(.playback)
        
        self.engine.attach(self.mixer)
        let stereoFormat =  AVAudioFormat(standardFormatWithSampleRate: self.engine.outputNode.outputFormat(forBus: 0).sampleRate, channels: 2)
        self.engine.connect(self.mixer, to: self.engine.outputNode, format: stereoFormat)
        self.engine.prepare()
        try! self.engine.start()
        
    }
    
    func play() {
        let audioPlayer = AVAudioPlayerNode()
        self.engine.attach(audioPlayer)
        let monoFormat =  AVAudioFormat(standardFormatWithSampleRate: self.engine.outputNode.outputFormat(forBus: 0).sampleRate, channels: 1)
        self.engine.connect(audioPlayer, to: self.mixer, format: monoFormat)
        
        // This file has to be in mono
        let url = Bundle.main.url(forResource: "your-mono-audio-file.mp3", withExtension: nil)
        let f = try! AVAudioFile(forReading: url!)
        audioPlayer.scheduleFile(f, at: nil, completionHandler: nil)
        audioPlayer.renderingAlgorithm = .HRTFHQ
        audioPlayer.position = AVAudio3DPoint(x: 20.0, y: 5.0, z: 0.0)
        audioPlayer.play()
    }
    
}

Answer 1

I want to be able to record audio and play back positional audio at the same time.

This is not possible over standard Bluetooth if you're both playing to and recording from the Bluetooth headsets. The option allowBluetooth does not mean "allow Bluetooth." It means "prefer HFP if available." (It's the worst named constant I know in Core Bluetooth.) HFP is a low-bandwidth bidirectional audio protocol designed for phone calls.

If you switch to the high-bandwidth audio protocol, A2DP, you'll find it's unidirectional, and so you cannot record from the microphone.

There is no widely-deployed Bluetooth protocol that gives you both high quality audio and access to a microphone. If you control the firmware, you can develop your own proprietary microphone audio stream over BLE (or iAP2 if it's a MFi device). But otherwise, there isn't a current solution.

I keep hoping that LEA will fix this, but I can't find any hint that it will. I also had hoped aptX might fix it (even though iPhones don't support it), but no luck there, either. I'm not certain why this use case isn't being worked on by the Bluetooth committee and vendors, but as best I know, nothing is on the horizon.