Solved - How to use "HEVC with Alpha channel" videos in SceneKit?

Question

At WWDC 2019, Apple announced that it was now handling HEVC with Alpha in iOS, and state that it could be used in SceneKit.

I've been trying for hours to make that happen, without any luck.

My goal is to use a video as a texture on a SCNNode plane, and to make this plane transparent where the video is transparent.

I use the HEVC with Alpha video provided by Apple in this xcode project (the video with 2 puppets).

I am developing with XCode 11.2 on a real device with iOS 13.2.

I tried different approaches.

Approach 1. Using a AVPlayer as diffuse + transparent contents of my SCNNode material

let videoURL = Bundle.main.url(forResource: “puppets_with_alpha_hevc“, withExtension: "mov", subdirectory: "Assets.scnassets");
let player = AVPlayer(url: videoURL! as URL)
let material = SCNMaterial()
material.diffuse.contents = player
material.transparent.contents = player
plane?.geometry?.materials = [material];
player.play();

Result : the video is rendered with black pixels instead of transparent ones.

Approach 2. Embed the AVPlayer in a SKVideoNode, then embed the SKVideoNode in a SKScene, and use the SKScene as a diffuse + material contents for the material of my SCNNode

This seems to be a common way to display video as material in SceneKit. Also, in the example provided by apple on how to render a “HEVC Video with alpha”, they use a SKVideoNode (but not in a SceneKit context).

let videoNode = SKVideoNode(avPlayer: player)
let spritescene = SKScene(size: CGSize(width: 360, height: 480))
let spriteKitScene = SKScene(size: CGSize(width: 360.0 / 2.0, height: 480.0 / 2.0))
spriteKitScene.scaleMode = .aspectFit
videoNode.position = CGPoint(x: spriteKitScene.size.width / 2.0, y: spriteKitScene.size.height / 2.0)
videoNode.size = spriteKitScene.size
spriteKitScene.addChild(videoNode)
let material = SCNMaterial()
material.diffuse.contents = spriteKitScene
material.transparent.contents = spriteKitScene
plane?.geometry?.materials = [material];

Result : get this crash:

validateFunctionArguments:3577: failed assertion `Fragment Function(FastSingle_FragFunc): missing sampler binding at index 0 for u_texture_sampler[0].'

Note : I don’t get this error when I use a regular HEVC video without an alpha channel.

Approach 3. Use a material shader to copy alpha values from the video to the transparent map of the material.

let material = SCNMaterial()
material.diffuse.contents = player
material.shaderModifiers = [
    SCNShaderModifierEntryPoint.surface :  "_surface.transparent.a = _surface.diffuse.a;"
];
plane?.geometry?.materials = [material];

Result : the video is rendered with black pixels instead of transparent ones. The alpha information does not seem to be in _surface.diffuse.a

Approach 4. Using the shader with the the SKScene instead of the AVPlayer

Result : same error than with approach 2.

Does anyone have a clue about how to make this work ?

Update : Tue, 07 Jan 2020

I tried my different approaches with iOS 13.3 and XCode 11.3. => Same results.

I just reported the bug to Apple, thanks for the link @mnuages.

Update : Tue, 24 Mar 2020

The problem is fixed with the release of iOS 13.4.

HEVC Videos with transparency are taken into account straight away in SceneKit, without having to mess with the transparent content of the material, or with shaders.

Answer 1

The first snippet of code should work (without having to set the transparent material property). If you make the following change in the macOS SceneKit game template you get the correct result, however it doesn't work with the iOS SceneKit game template.

// retrieve the ship node
let ship = scene.rootNode.childNode(withName: "ship", recursively: true)!
let videoURL = Bundle.main.url(forResource: "puppets_with_alpha_hevc", withExtension: "mov", subdirectory: "art.scnassets");
let player = AVPlayer(url: videoURL! as URL)
let material = SCNMaterial()
material.diffuse.contents = player
player.play();
ship.enumerateHierarchy { (node, _) in
    if let geometry = node.geometry {
        geometry.materials = [material];
    }
}

This a case where one might want to file a bug for the iOS version to be fixed.