XMVector3Project unexpected behaviour

Question

I'm trying to figure out World space to Screen space transform. As I understand, in D3D11, function XMVector3Project should handle this. However, when I use it like this:

XMVECTOR eye = XMVectorSet(10000, 0.0f, 1.5f, 0.0f);
XMVECTOR at = XMVectorSet(10000, 0.0f, 0.0f, 0.0f);
XMVECTOR up = XMVectorSet(0.0f, 1.0f, 0.0f, 0.0f);



auto viewMatrix = XMMatrixTranspose(XMMatrixLookAtRH(eye2, at2, up2));
XMVECTOR vec = XMVector3Project(XMVectorSet(0.0, 0.0, 0.0, 1.0f), 0, 0, 480, 800, 0, 1, XMMatrixIdentity(), viewMatrix, XMMatrixIdentity());

it returns point (240, 480). I don't understand how that's possible, cause even with no Projection matrix, when I set view matrix to show point (1000, 1000, x), Point (0,0,0) shouldn't show on screen at all. That's just my view, probably wrong, so I would like to know how is that intended behaviour?

Answer 1

I think the problem here is your use of XMMatrixTranspose. DirectXMath (aka XNAMath version 3 aka xboxmath) functions are all written assuming you have row-major matrices either left-handed or right-handed. By applying the XMMatrixTranspose to the lookat matrix, you are making it column-major. While this is commonly done as a last step before setting it into a Constant Buffer for consumption by HLSL (see MSDN DirectXMath Programmer's Guide and MSDN HLSL docs for details), the result doesn't make sense to use this way with XMVector3Project.

BTW, I'm assuming your use of XMVectorSet here is just for testing, but the efficient way to code a constant XMVECTOR is using XMVECTORF32.

static const XMVECTORF32 eye = { 10000, 0.0f, 1.5f, 0.0f };
static const XMVECTORF32 at = { 10000, 0.0f, 0.0f, 0.0f };
static const XMVECTORF32 up = { 1.0f, 0.0f, 0.0f, 0.0f };