How to draw camera video as background on IOS using Swift scenekit?

Question

I am trying to develop an augmented reality app using swift and scenekit on ios. Is there a way to draw the video captured by the device camera as a background of the scene?

Answer 1

This worked for me,

I used AVFoundation to capture the video input of the device camera:

   let captureSession = AVCaptureSession()
   let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
    previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill

    if let videoDevice = AVCaptureDevice.defaultDeviceWithMediaType(AVMediaTypeVideo) {
        var err: NSError? = nil
        if let videoIn : AVCaptureDeviceInput = AVCaptureDeviceInput.deviceInputWithDevice(videoDevice, error: &err) as? AVCaptureDeviceInput {
            if(err == nil){
                if (captureSession.canAddInput(videoIn as AVCaptureInput)){
                    captureSession.addInput(videoIn as AVCaptureDeviceInput)
                }
                else {
                    println("Failed to add video input.")
                }
            }
            else {
                println("Failed to create video input.")
            }
        }
        else {
            println("Failed to create video capture device.")
        }
    }
    captureSession.startRunning()

At this point, based on Apple's documentation of the background property of SCNScene, I expected to add the instance of AVCaptureVideoPreviewLayer to an SCNScene's background.contents, with something like:

   previewLayer.frame = sceneView.bounds
   sceneView.scene.background.contents = previewLayer 

This changed the background color of my scene from default white to black, but offered no video input. This may be an iOS bug?. So Plan B. Instead, I added the 'AVCaptureVideoPreviewLayer' as a sublayer of a UIView's layer:

 previewLayer.frame = self.view.bounds
 self.view.layer.addSublayer(previewLayer)

I then set an SCNView as a subview of that same UIView, setting the SCNView's background color to clear:

    let sceneView = SCNView()
    sceneView.frame = self.view.bounds
    sceneView.backgroundColor = UIColor.clearColor()
    self.view.addSubview(sceneView)

The device camera's video is now visible as the background of the scene.

I've created a small demo.

Answer 2

As of iOS 11, you can now use an AVCaptureDevice as a material on an object or the scene's background (see "Using Animated Content" here)

Example:

 
    let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back)!
    scnScene.background.contents = captureDevice

Answer 3

Yes, you can use a AVCaptureVideoPreviewLayer as the contents of a material property (just make sure that you give the layer a bounds).

The scene view has a material property for the background that you can assign the video preview to (assign the layer to the background's contents).

var background: SCNMaterialProperty! { get }