How do I reference an arbitrary id<MTLTexture> for use in a SCNTechnique?

Question

TL;DR:

How does one reference a 'not on disk' sampler2D symbol and pass it to an SCNTechnique? My technique works if I reference an image from my bundle, but if I don't, I cannot find a way to pass in an existing id<MTLTexture> to the sampler symbol my technique has set up.

Long:

I have a valid working SCNTechnique which uses a custom sampler2D symbol as an input to my metal fragment pass. I am attempting to pass in an external (not from Scenekit) id<MTLTexture> that I get from a hardware sensor as input in a post process pass.

Following the SCNShadable docs which state an id<MTLTexture> can be passed as a shader input via an SCNMaterialProperty which has the proper contents set. This 100% works in a shader modifier pass - but fails for SCNTecnique!

let texture = CVMetalTextureGetTexture(textureRef!)

if self.material == nil
{
    self.material = SCNMaterialProperty(contents:texture)
}
else
{
    self.material?.contents = texture
}

self.currentTechnique?.setObject(self.material, forKeyedSubscript: "myTextureSamplerSymbol" as NSCopying)

For a SCNTechnique, I get error logs indicating "No Storage for texture" and the Metal GPU frame capture indicates there is a default 4x4 pixel white texture set for the sampler (presumably from the SCNTecnique?). However, I've been able to validate that my custom id<MTLTexture> is valid and has contents in the debugger - its format, width, height and contents all are as expected, I just can't seem to reference an arbitrary texture into a scene kit technique pass correctly.

If I declare my symbol in the SCNTechnique plist file to reference an image like so:

<dict>
    <key>type</key>
    <string>sampler2D</string>
    <key>image</key>
    <string>star.png</string>
</dict>

And my pass inputs like so:

<dict>
    <key>colorSampler</key>
    <string>COLOR</string>
    <key>depthSampler</key>
    <string>DEPTH</string>
    <key> myTextureSampler</key>
    <dict>
        <key>target</key>
        <string> myTextureSamplerSymbol </string>
    </dict>
</dict>

Then my pass works and the star.png texture referenced.

Has anyone gotten something like this to work?

Thank you.

Answer 1

Fairly certain I got this working.

Swift code to setup the MTLTexture and set it into the SCNTechnique.

let tech:SCNTechnique = getTechnique()

let textureLoader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!)
let filePath = Bundle.main.url(forResource: "gradient", withExtension: "png")!

do {
    let gradTexture: MTLTexture = try textureLoader.newTexture(URL: filePath, options: nil)
    let matPropTexture = SCNMaterialProperty(contents: gradTexture)
    tech.setObject(matPropTexture, forKeyedSubscript: "myTexture" as NSCopying)
} catch {
    print("Unexpected error: \(error).")
}

scnView.technique = tech

gradient.png is a 256 x 1px color gradient (blue -> green -> red) image I use to map a single value to a pseudo colour. (eg; )

Here's a complete technique pass definition from the passes dict in my plist.

<key>mix_outline</key>
<dict>
    <key>draw</key>
    <string>DRAW_QUAD</string>
    <key>metalVertexShader</key>
    <string>pass_through_vertex</string>
    <key>metalFragmentShader</key>
    <string>mix_outline_fragment</string>
    <key>inputs</key>
    <dict>
        <key>colorSampler</key>
        <string>color_scene</string>
        <key>depthSampler</key>
        <string>depth_outline</string>
        <key>myTextureSampler</key>
        <string>myTexture</string>
    </dict>
    <key>outputs</key>
    <dict>
        <key>color</key>
        <string>COLOR</string>
    </dict>
</dict>

myTexture needs to be defined in the symbols section of the technique plist also.

<key>symbols</key>
<dict>
    <key>myTexture</key>
    <dict>
        <key>type</key>
        <string>sampler2D</string>
    </dict>
</dict>

I too saw the "pass has no storage for input myTextureSampler" error message when this symbols block was not included. This may be your problem?

And lastly the fragment shader definition.

fragment half4 mix_outline_fragment(out_vertex_t vert [[stage_in]],
                                    texture2d<float, access::sample> colorSampler [[texture(0)]],
                                    texture2d<float, access::sample> depthSampler [[texture(1)]],
                                    texture2d<float, access::sample> myTextureSampler [[texture(2)]])
{
    float4 myTextureColor = myTextureSampler.read(uint2(55, 0));
    float4 outline = depthSampler.sample( s, vert.uv);
    float4 scene_color = colorSampler.sample( s, vert.uv);
    // float4 fragment_color = mix(scene_color, float4(0.0, 0.0, 0.0, 0.0), outline.r);
    float4 fragment_color = mix(scene_color, myTextureColor, outline.r);
    return half4(fragment_color);
}

There's a few other passes involved in this technique that I haven't included; just to give some context it renders the scene in one pass and uses the output depth buffer in another pass with a Sobel operator to generate edges into the depthSampler texture. The above pass + fragment shader draws these edges on top of the original rendering of the scene. I was using solid black for the edges, but after looking into this it seems I'm able to read a color out of the MTLTexture I provide to the SCNTechnique and use it for the edges.