IOS Metal Stencil Buffer project crashes when Metal Validation Enabled

Question

I am trying to learn how to implement Stencil Buffer in Metal IOS. For now It works fine without Metal Validation. But as soon as I turn on Metal Validations, project crashes.

Here is my code:- Metal View:-

import MetalKit

class DrawingView: MTKView {

    //Renderer to handle all the rendering of the Drawing View
    public var renderer: Renderer!

    override init(frame frameRect: CGRect, device: MTLDevice?) {
        super.init(frame: frameRect, device: device)
        self.configDrawingView()
    }

    required init(coder: NSCoder) {
        super.init(coder: coder)
        self.configDrawingView()
    }

    private func configDrawingView() {
        //Setting system default GPU
        self.device = MTLCreateSystemDefaultDevice()
        //Setting pixel format for the view
        self.colorPixelFormat = MTLPixelFormat.bgra8Unorm
        //Setting clear color of the view. View will be cleared in every frame
        self.clearColor = MTLClearColorMake(1.0, 1.0, 1.0, 1.0)
        //Setting sample count of drawing textures
        self.sampleCount = 1
        //Setting prefered frame rate
        self.preferredFramesPerSecond = 120
        //Setting auto resizing of the drawable to false. If the value is true, the currentDrawable object’s texture, depthStencilTexture object, and multisampleColorTexture object automatically resize as the view resizes. If the value is false, drawableSize does not change and neither does the size of these objects.
        self.autoResizeDrawable = false

        //Stencil
        self.clearStencil = 0
        self.depthStencilPixelFormat = MTLPixelFormat.depth32Float_stencil8

        //Setting up Renderer
        self.renderer = Renderer(withDevice: self.device!)
        //Using renderer and the MTKView delegate. This will call draw method every frame
        self.delegate = renderer
    }

}

Renderer:-

import MetalKit

class Renderer: NSObject {

    //Device
    private var device: MTLDevice!

    //CommandQueue
    private var commandQueue: MTLCommandQueue!

    //Mirror Vertex
    private var paintData: [Vertex] = [
        Vertex(position: float4(-0.75, -0.75, 0, 1), color: float4(1.0, 1.0, 0.0, 1.0)),
        Vertex(position: float4(-0.75, 0.75, 0, 1), color: float4(1.0, 1.0, 0.0, 1.0)),
        Vertex(position: float4(0.75, 0.75, 0, 1), color: float4(1.0, 1.0, 0.0, 1.0)),
        Vertex(position: float4(0.75, -0.75, 0, 1), color: float4(1.0, 1.0, 0.0, 1.0))
    ]
    //Mirror Indices
    private var paintIndicies: [UInt32] = [0, 1, 2, 0, 2, 3]
    //Mirror Vertex Buffer
    private var paintVertexBuffer: MTLBuffer!
    //Mirror Index Buffer
    private var paintIndexBuffer: MTLBuffer!

    //Paint Vertex
    private var stencilData: [Vertex] = [
        Vertex(position: float4(-1, 0, 0, 1), color: float4(0.0, 0.0, 1.0, 1.0)),
        Vertex(position: float4(0, 1, 0, 1), color: float4(0.0, 0.0, 1.0, 1.0)),
        Vertex(position: float4(0, -1, 0, 1), color: float4(0.0, 0.0, 1.0, 1.0)),
        Vertex(position: float4(1, 0, 0, 1), color: float4(0.0, 0.0, 1.0, 1.0))
    ]
    //Paint indices
    private var stencilIndices: [UInt32] = [0, 1, 2, 2, 1, 3]
    //Paint Vertex Buffer
    private var stencilVertexBuffer: MTLBuffer!
    //Paint Index Buffer
    private var stencilIndexBuffer: MTLBuffer!

    //Depth Stencil State
    private var depthStencilState: MTLDepthStencilState!
    private var maskStencilState: MTLDepthStencilState!
    private var drawStencilState: MTLDepthStencilState!

    private var library: MTLLibrary!

    private var vertexFunction: MTLFunction!
    private var fragmentFunction: MTLFunction!
    private var stencilFragmentFunction: MTLFunction!

    private var pipelineState: MTLRenderPipelineState!
    private var stencilPipelineState: MTLRenderPipelineState!

    init(withDevice device: MTLDevice) {
        super.init()
        self.device = device
        self.configure()
    }

    private func configure() {
        self.commandQueue = self.device.makeCommandQueue()

        self.setupPipelines()
        self.setupDepthStencil()
        self.setupBuffers()
    }

    private func setupDepthStencil() {
        var depthStencilStateDescriptor: MTLDepthStencilDescriptor = MTLDepthStencilDescriptor()
        depthStencilStateDescriptor.depthCompareFunction = MTLCompareFunction.less
        depthStencilStateDescriptor.isDepthWriteEnabled = true

        self.depthStencilState = self.device.makeDepthStencilState(descriptor: depthStencilStateDescriptor)

        depthStencilStateDescriptor = MTLDepthStencilDescriptor()
        depthStencilStateDescriptor.depthCompareFunction = MTLCompareFunction.less

        var stencilDescriptor: MTLStencilDescriptor = MTLStencilDescriptor()
        stencilDescriptor.stencilCompareFunction = MTLCompareFunction.always
        stencilDescriptor.depthStencilPassOperation = MTLStencilOperation.replace
        depthStencilStateDescriptor.backFaceStencil = stencilDescriptor
        depthStencilStateDescriptor.frontFaceStencil = stencilDescriptor
        depthStencilStateDescriptor.isDepthWriteEnabled = false

        self.drawStencilState = self.device.makeDepthStencilState(descriptor: depthStencilStateDescriptor)

        depthStencilStateDescriptor = MTLDepthStencilDescriptor()
        depthStencilStateDescriptor.depthCompareFunction = MTLCompareFunction.less

        stencilDescriptor = MTLStencilDescriptor()
        stencilDescriptor.stencilCompareFunction = MTLCompareFunction.equal
        depthStencilStateDescriptor.frontFaceStencil = stencilDescriptor
        depthStencilStateDescriptor.backFaceStencil = stencilDescriptor
        depthStencilStateDescriptor.isDepthWriteEnabled = true

        self.maskStencilState = self.device.makeDepthStencilState(descriptor: depthStencilStateDescriptor)

    }

    private func setupPipelines() {
        self.library = self.device.makeDefaultLibrary()

        self.vertexFunction = self.library.makeFunction(name: "vertexShader")
        self.fragmentFunction = self.library.makeFunction(name: "fragmentShader")
        self.stencilFragmentFunction = self.library.makeFunction(name: "stencilFragmentShader")

        var renderPipelineDescriptor: MTLRenderPipelineDescriptor = MTLRenderPipelineDescriptor()
        renderPipelineDescriptor.vertexFunction = self.vertexFunction
        renderPipelineDescriptor.fragmentFunction = self.fragmentFunction
        renderPipelineDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormat.bgra8Unorm
        renderPipelineDescriptor.depthAttachmentPixelFormat = MTLPixelFormat.depth32Float_stencil8
        renderPipelineDescriptor.stencilAttachmentPixelFormat = MTLPixelFormat.depth32Float_stencil8

        do {
            self.pipelineState = try self.device.makeRenderPipelineState(descriptor: renderPipelineDescriptor)
        } catch {
            print("\(error)")
        }

        var renderPipelineDescriptorNoDraw: MTLRenderPipelineDescriptor = MTLRenderPipelineDescriptor()
        renderPipelineDescriptorNoDraw.vertexFunction = self.vertexFunction
        renderPipelineDescriptorNoDraw.fragmentFunction = self.stencilFragmentFunction
        renderPipelineDescriptorNoDraw.depthAttachmentPixelFormat = MTLPixelFormat.depth32Float_stencil8
        renderPipelineDescriptorNoDraw.stencilAttachmentPixelFormat = MTLPixelFormat.depth32Float_stencil8
        renderPipelineDescriptorNoDraw.colorAttachments[0].pixelFormat = MTLPixelFormat.bgra8Unorm //Problem Here ---- (1)
        renderPipelineDescriptorNoDraw.colorAttachments[0].writeMask = []

        do {
            self.stencilPipelineState = try self.device.makeRenderPipelineState(descriptor: renderPipelineDescriptorNoDraw)
        } catch {
            print("\(error)")
        }
    }

    private func setupBuffers() {
        self.stencilVertexBuffer = self.device.makeBuffer(bytes: self.stencilData, length: MemoryLayout<Vertex>.stride * self.stencilData.count, options: [])
        self.stencilIndexBuffer = self.device.makeBuffer(bytes: self.stencilIndices, length: MemoryLayout<UInt32>.size * self.stencilIndices.count, options: [])

        self.paintVertexBuffer = self.device.makeBuffer(bytes: self.paintData, length: MemoryLayout<Vertex>.stride * self.paintData.count, options: [])
        self.paintIndexBuffer = self.device.makeBuffer(bytes: self.paintIndicies, length: MemoryLayout<UInt32>.size * self.paintIndicies.count, options: [])
    }

}

extension Renderer: MTKViewDelegate {
    func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {

    }

    func draw(in view: MTKView) {
        let buffer: MTLCommandBuffer = self.commandQueue.makeCommandBuffer()!


        let renderPassDescriptor = view.currentRenderPassDescriptor
        renderPassDescriptor!.colorAttachments[0].clearColor = MTLClearColorMake(1.0, 1.0, 1.0, 1.0)
        let encoderClear = buffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor!)
        encoderClear!.endEncoding()

        let renderPassDescriptorOther = MTLRenderPassDescriptor()
        renderPassDescriptorOther.colorAttachments[0].loadAction = MTLLoadAction.load
        renderPassDescriptorOther.colorAttachments[0].storeAction = MTLStoreAction.store
        renderPassDescriptorOther.colorAttachments[0].texture = view.currentDrawable?.texture
        let commandEncoderOther = buffer.makeRenderCommandEncoder(descriptor: renderPassDescriptorOther)
        Plane(withDevice: self.device, sizeOf: float2(100, 100), andPlaneCenterTo: float2(100, 100), withColor: float4(1.0, 0.0, 0.0, 1.0)).render(commandEncoder: commandEncoderOther!)
        commandEncoderOther?.endEncoding()

        let renderPassDescriptorOther1 = MTLRenderPassDescriptor()
        renderPassDescriptorOther1.colorAttachments[0].loadAction = MTLLoadAction.load
        renderPassDescriptorOther1.colorAttachments[0].storeAction = MTLStoreAction.store
        renderPassDescriptorOther1.colorAttachments[0].texture = view.currentDrawable?.texture
        let encoder: MTLRenderCommandEncoder = buffer.makeRenderCommandEncoder(descriptor: renderPassDescriptorOther1)!

        //Stencil
        encoder.setRenderPipelineState(self.stencilPipelineState)
        encoder.setStencilReferenceValue(1)
        encoder.setDepthStencilState(self.drawStencilState)
        encoder.setVertexBuffer(self.stencilVertexBuffer, offset: 0, index: 0)
        encoder.drawIndexedPrimitives(type: MTLPrimitiveType.triangle, indexCount: self.stencilIndices.count, indexType: MTLIndexType.uint32, indexBuffer: self.stencilIndexBuffer, indexBufferOffset: 0)

        //Paint
        encoder.setRenderPipelineState(self.pipelineState)
        encoder.setDepthStencilState(self.maskStencilState)
        encoder.setVertexBuffer(self.paintVertexBuffer, offset: 0, index: 0)
        encoder.drawIndexedPrimitives(type: MTLPrimitiveType.triangle, indexCount: self.paintIndicies.count, indexType: MTLIndexType.uint32, indexBuffer: self.paintIndexBuffer, indexBufferOffset: 0)
        encoder.endEncoding()

        buffer.present(view.currentDrawable!)
        buffer.commit()
    }
}

Here is my shader file:-

#include <metal_stdlib>
using namespace metal;

struct Vertex {
    float4 position [[position]];
    float4 color;
};  

vertex Vertex vertexShader(const device Vertex *vertexArray [[buffer(0)]], unsigned int vid [[vertex_id]]) {
    return vertexArray[vid];
}

fragment float4 fragmentShader(Vertex interpolated [[stage_in]]) {
    return interpolated.color;
}

fragment float4 fragmentShader_stencil(Vertex v [[stage_in]])
{
    return float4(1, 1, 1, 0.0);
}

When I turn on Metal Validation, It gives me this error:-

[MTLDebugRenderCommandEncoder validateFramebufferWithRenderPipelineState:]:1236:
failed assertion `For depth attachment, the renderPipelineState pixelFormat 
must be MTLPixelFormatInvalid, as no texture is set.'

Then I changed (1) pixel format in Renderer to MTLPixelFormat.inavlid it gives me another error :-

[MTLDebugRenderCommandEncoder validateFramebufferWithRenderPipelineState:]:1196: 
failed assertion `For color attachment 0, the render pipeline's pixelFormat 
(MTLPixelFormatInvalid) does not match the framebuffer's pixelFormat 
(MTLPixelFormatBGRA8Unorm).'

Is there way to fix this. I want Metal Validation to be enabled. With Validation disabled it works fine.

Answer 1

Since you're manually creating render pass descriptors, you need to ensure that the textures and load/store actions of all relevant attachments are configured.

You've specified to your MTKView that it should manage a depth/stencil texture for you, and configured your render pipeline state to expect a depth/stencil texture, so you need to provide such a texture when creating your render pass descriptor:

renderPassDescriptorOther1.depthAttachment.loadAction = .clear
renderPassDescriptorOther1.depthAttachment.storeAction = .dontCare
renderPassDescriptorOther1.depthAttachment.texture = view.depthStencilTexture

renderPassDescriptorOther1.stencilAttachment.loadAction = .clear
renderPassDescriptorOther1.stencilAttachment.storeAction = .dontCare
renderPassDescriptorOther1.stencilAttachment.texture = view.depthStencilTexture

Incidentally, I don't see the reason for running a separate "clear" pass before drawing. It seems that you could just set the loadAction of renderPassDescriptorOther's color attachment to .clear.