How to enable Hardware Percentage Closer Filtering?

Question

I am trying to implement PCF filtering to my shaow maps, and so modified the GPU Gems article ( http://http.developer.nvidia.com/GPUGems/gpugems_ch11.html ) so that it could be run on current shaders. The modification includes replacing tex2Dproj function with Texture2D.Sample() function so it accepts Sampler States which are created in DirectX11. Then I compared the offset values with a normal shadow map comparison:

float2 ShTex;
ShTex.x = PSIn.ShadowMapSamplingPos.x/PSIn.ShadowMapSamplingPos.w/2.0f +0.5f;
ShTex.y = -PSIn.ShadowMapSamplingPos.y/PSIn.ShadowMapSamplingPos.w/2.0f +0.5f;
float realDistance = PSIn.ShadowMapSamplingPos.z/PSIn.ShadowMapSamplingPos.w;


(realDistance - xShBias) <= offset_lookup(xTextureSh, texSampler, ShTex, float2(x, y), scale); //Blocker-receiver comparison (the usual stuff, only that it gets 4*4 samples from near offsets then divides the final value with 16)

Now what I get in return is this:

Where in the article it says and shows that the hardware should interpolate the results magically. I guess the tex2Dproj may do that, but no success in getting it to work.

Lately I came across SampleCmpLevelZero function which I tried out with this sampler state:

SamplerComparisonState cmpSampler
{
   Filter = COMPARISON_MIN_MAG_MIP_LINEAR;
   AddressU = Clamp;
   AddressV = Clamp;

   ComparisonFunc = LESS;
};

When I get it to work, the entire scene randomly flickers from shadowed to unshadowed and partly shadowed (which seemed like the correct shadows).

xTextureSh.SampleCmpLevelZero( cmpSampler, ShTex.xy, realDistance - xShBias );

Sadly, I have found someone with the same proble but who managed to fix it by rewriting the filter to the mode I already have applied.

My Depth texture is in the following format: DXGI_FORMAT_R24_UNORM_X8_TYPELESS

I am now in great confusion by this mystery regarding correct PCF filtering, is there anyone who can suggest anything at all? Thank you in advance and for reading through the end.

Answer 1

Okay, so I have completed it. By using SampleCmpLevelZero, I have tried to declare a SamplerComparisonState inside the HLSL code, but it turned out to be malfunctioning. The shadow map bilinear filtering did not work. Then I tried to create a Sampler state inside the D3D11 API, setting D3D11_FILTER_COMPARISON_MIN_MAG_LINEAR_MIP_POINT as the filter of the description, D3D11_COMPARISON_LESS for the Comparison Function of the description. Inside the shader I declared a SamplerComparisonState instead of the usual SamplerState, and have not filled out the description there, but registered it from the api:

SamplerComparisonState compSampler:register(s2);

Then the magic finally kicked in:

As you can see, the result is a seamless penumbra around the shadow instead of the staircases. Even without the manual sampling of the previous picture, We can achieve quite acceptable results with one manual sample (and more automatic samples done by SampleCmpLevelZero):

Sadly, I do not know what caused the malfunctioning of the sampler declaration inside the shader and I do not like that. If anyone has experienced anyting like that, I would be glad if he/she shader it with me.

Also thanks to MHGameWork for trying to help me solve this problem.

UPDATE: Since a lot of time passed since the question was asked, the issue of declaring samplers in shaders became known to me in the meantime, so I am putting it here: You can only declare sampler objects inside a shader if you are using the Effects framework. When using barebones shaders, they need to be uploaded by API calls.