VNTrackRectangleRequest internal error

Question

I'm trying to get a simple rectangle tracking controller going, and I can get rectangle detection going just fine, but the tracking request always ends up failing for a reason I can't quite find.

Sometimes the tracking request will fire it's callback a few times before failing, other times it fails immediately before a single callback occurs. I feel it's something to do with how I submit the requests but I can't get to the bottom of it.

Here's the code for the view controller

class TestController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {

    // Video capture
    private var videoSession = AVCaptureSession()
    private var videoLayer: AVCaptureVideoPreviewLayer!

    // Detection
    private var detectionRequest: VNDetectRectanglesRequest?
    private let sequenceHandler = VNSequenceRequestHandler()

    // Tracking
    private var trackingRequest: VNTrackRectangleRequest?
    private var shape: Detection?
    private var pixelBuffer: CVImageBuffer?




    // MARK: Setup
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        startVideoFeed()
    }

    override func viewDidLayoutSubviews() {
        videoLayer.frame = view.layer.bounds
    }

    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        startDetectingRectangles()
    }



    private func startDetectingRectangles() {
        let request = VNDetectRectanglesRequest(completionHandler: didDetectRectangle)
        request.maximumObservations = 1
        request.minimumSize = 0.07
        request.minimumConfidence = 0.9
        request.minimumAspectRatio = 0.5
        request.maximumAspectRatio = 2
        request.quadratureTolerance = 10
        detectionRequest = request
    }

    private func didDetectRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observations = request.results, observations.count > 0 else { return }
        let results = observations.map { $0 as? VNRectangleObservation }

        for case let rectangle? in results {
            detectionRequest = nil

            let request = VNTrackRectangleRequest(rectangleObservation: rectangle, completionHandler: didTrackRectangle)
            trackingRequest = request
        }
    }


    private func didTrackRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observation = request.results?.first as? VNRectangleObservation else { return }

        // Create or update UI
    }




    // Start capturing video frames
    private func startVideoFeed() {

        // Session config
        videoSession.sessionPreset = .photo

        // Create device and input to device
        guard
            let captureDevice = AVCaptureDevice.default(for: .video),
            let deviceInput = try? AVCaptureDeviceInput(device: captureDevice)
            else {
                fatalError("Error setting up capture device.")
        }

        // Setup device output
        let deviceOutput = AVCaptureVideoDataOutput()
        deviceOutput.setSampleBufferDelegate(self, queue: DispatchQueue.global(qos: .default))

        // Set input and output
        videoSession.addInput(deviceInput)
        videoSession.addOutput(deviceOutput)

        // Setup video display layer
        videoLayer = AVCaptureVideoPreviewLayer(session: videoSession)
        videoLayer.frame = view.bounds
        videoLayer.videoGravity = .resizeAspectFill
        view.layer.addSublayer(videoLayer)

        videoSession.startRunning()
    }

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

        pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)

        guard let pixelBuffer = pixelBuffer else { return }
        do {

            var requests: [VNRequest] = []

            if let detectionRequest = detectionRequest {
                requests.append(detectionRequest) }
            if let trackingRequest = trackingRequest {
                requests.append(trackingRequest) }

            try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right)
        } catch {
            print(error)
        }
    }
}

The error is printing:

Error Domain=com.apple.vis Code=9 "Internal error: Tracking of one of the corners failed, confidence = 0.000000; threshold = 0.650000" UserInfo={NSLocalizedDescription=Internal error: Tracking of one of the corners failed, confidence = 0.000000; threshold = 0.650000}

This is being output every frame after the tracking request first fails, and the request never recovers itself. It's the try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right) line that is throwing the error (which I then print).

I've looked at the object observation examples dotted around the web and they seem to require a new tracking request to be made each frame to track the object. I've tried creating a new VNTrackRectangleRequest in the didTrackRectangle function so each frame has a new request, but I get the same issue.

Any help regarding this is really appreciated.

Answer 1

I was seeing this error every frame, and to fix it I found that I needed to call performRequests:onCVPixelBuffer:orientation:error: with the correct orientation.

(For portrait orientation with the rear-facing camera, I believe the orientation is kCGImagePropertyOrientationRight).

From the documentation:

For each such request, call the sequence request handler’s performRequests:onCVPixelBuffer:orientation:error: method, making sure to pass in the video reader’s orientation to ensure upright tracking.