How to delegate ARSession and AVCaptureSession at the same time?

Question

I've seen people running ARKit with ARSCNView, and they're able to fetch said a QR code rectangle in the camera and render something at the position somehow related to the place QR code resided in the space. I thought they did it by delegating AVCaptureMetadataOutputObjectsDelegate, so I went ahead and attempt to achieve the following logic:

1. Running ARSCNViewDelegate, ARSessionDelegate and AVCaptureMetadataOutputObjectsDelegate at the same time.

2. Wire up an ARSCNView, delegate the view and its session to self.

3. Wire up an AVCaptureSession, delegate it to self, and start the session.

4. Start the ARSCNView session, since the official example did this in viewWillAppear(), I did the same thing.

The application did run without any error, but I could only get a few callbacks from AVCaptureMetadataOutputObjectsDelegate, like 3 to 5, and then it would never be called again like the delegate of ARSession seized its authority, was I implementing the wrong approach or you just can't delegate ARSession and AVCaptureSession at the same time?

Answer 1

I did exactly what you describe and have the same problems. It seems that ARKit relies upon the AVCapture system, and doesn't support using more than one capture device at a time. I found two solutions.

1. Start an ARSession, and implement the function session(_:didUpdate:) of ARSessionDelegate. Every time you capture an ARFrame search for a QR code in the image of the frame. (code)

   func session(_ session: ARSession, didUpdate frame: ARFrame) {
       let image = CIImage(cvPixelBuffer: frame.capturedImage)
       let detector = CIDetector(ofType: CIDetectorTypeQRCode, context: nil, options: nil)
       let features = detector!.features(in: image)
   
       for feature in features as! [CIQRCodeFeature] {
           if !discoveredQRCodes.contains(feature.messageString!) {
               discoveredQRCodes.append(feature.messageString!)
               let url = URL(string: feature.messageString!)
               let position = SCNVector3(frame.camera.transform.columns.3.x,
                                         frame.camera.transform.columns.3.y,
                                         frame.camera.transform.columns.3.z)
           }
        }
   } 
   

2. Start an AVCaptureSession, when you identify and decode your QR code stop it and start an ARSession. (not recommended)