Kinect SDK: align depth and color frames

Question

I'm working with Kinect sensor and I'm trying to align depth and color frames so that I can save them as images which "fit" into each other. I've spent a lot of time going through msdn forums and modest documentation of Kinect SDK and I'm getting absolutely nowhere.

Based on this answer: Kinect: Converting from RGB Coordinates to Depth Coordinates

I have the following function, where depthData and colorData are obtained from NUI_LOCKED_RECT.pBits and mappedData is the output containing new color frame, mapped to depth coordinates:

bool mapColorFrameToDepthFrame(unsigned char *depthData, unsigned char* colorData, unsigned char* mappedData)
{
    INuiCoordinateMapper* coordMapper;

    // Get coordinate mapper
    m_pSensor->NuiGetCoordinateMapper(&coordMapper);

    NUI_DEPTH_IMAGE_POINT* depthPoints = new NUI_DEPTH_IMAGE_POINT[640 * 480];

    HRESULT result = coordMapper->MapColorFrameToDepthFrame(NUI_IMAGE_TYPE_COLOR, NUI_IMAGE_RESOLUTION_640x480, NUI_IMAGE_RESOLUTION_640x480, 640 * 480, reinterpret_cast<NUI_DEPTH_IMAGE_PIXEL*>(depthData), 640 * 480, depthPoints);
    if (FAILED(result))
    {
        return false;
    }    

    int pos = 0;
    int* colorRun = reinterpret_cast<int*>(colorData);
    int* mappedRun = reinterpret_cast<int*>(mappedData);

    // For each pixel of new color frame
    for (int i = 0; i < 640 * 480; ++i)
    {
        // Find the corresponding pixel in original color frame from depthPoints
        pos = (depthPoints[i].y * 640) + depthPoints[i].x;

        // Set pixel value if it's within frame boundaries
        if (pos < 640 * 480)
        {
            mappedRun[i] = colorRun[pos];
        }
    }

    return true;
}

All I get when running this code is an unchanged color frame with removed (white) all pixels where depthFrame had no information.

Answer 1

With the OpenNI framework there an option call registration.

IMAGE_REGISTRATION_DEPTH_TO_IMAGE – The depth image is transformed to have the same apparent vantage point as the RGB image.

OpenNI 2.0 and Nite 2.0 works very well to capture Kinect information and there a lot of tutorials.

You can have a look to this :

Kinect with OpenNI

And OpenNi have a example in SimplerViewer that merge Depth and Color maybe you can just look on that and try it.