Kinect Depth and Image Frames Alignment

Question

I am playing around with new Kinect SDK v1.0.3.190. (other related questions in stackoverflow are on previous sdk of kinect) I get depth and color streams from Kinect. As the depth and RGB streams are captured with different sensors there is a misalignment between two frames as can be seen below.

Only RGB

Only Depth

Depth & RGB

I need to align them and there is a function named MapDepthToColorImagePoint exactly for this purpose. However it doesn't seem to work. here is a equally blended (depth and mapped color) result below which is created with the following code

            Parallel.For(0, this.depthFrameData.Length, i =>
        {
            int depthVal = this.depthFrameData[i] >> 3;
            ColorImagePoint point = this.kinectSensor.MapDepthToColorImagePoint(DepthImageFormat.Resolution640x480Fps30, i / 640, i % 640, (short)depthVal, ColorImageFormat.RgbResolution640x480Fps30);
            int baseIndex = Math.Max(0, Math.Min(this.videoBitmapData.Length - 4, (point.Y * 640 + point.X) * 4));

            this.mappedBitmapData[baseIndex] = (byte)((this.videoBitmapData[baseIndex]));
            this.mappedBitmapData[baseIndex + 1] = (byte)((this.videoBitmapData[baseIndex + 1]));
            this.mappedBitmapData[baseIndex + 2] = (byte)((this.videoBitmapData[baseIndex + 2]));
        });

where

depthFrameData -> raw depth data (short array)

videoBitmapData -> raw image data (byte array)

mappedBitmapData -> expected result data (byte array)

order of the parameters, resolution, array sizes are correct (double checked).

The result of the code is:

The misalignment continues! What is even worse is that, result image after using MapDepthToColorImagePoint is exactly the same with the original image.

Would be appreciated if someone could help me to find out my mistake or at least explain me what is MapDepthToColorImagePoint for (assuming that I misunderstood its functionality)?

Answer 1

This will always happen slightly because the two sensors are mounted at slightly different places.

Try it:

Look at some object with your two eyes, then try using only your left eye, then only your right eye. Things look slightly different because your two eyes are not in exactly the same place.

However: it is possible to correct a lot of the issues with some of the API codes.

I'm using Kinect for Windows 1.5, so the API's are slightly different from the 1.0.

short[] depth=new short[320*240];
// fill depth with the kinect data
ColorImagePoint[] colorPoints=new ColorImagePoint[320*240];
// convert mappings
kinect.MapDepthFrameToColorFrame(DepthImageFormat.Resolution320x240Fps30,
            depth, ColorImageFormat.RgbResolution640x480Fps30, colorPoints);
// now do something with it
for(int i=0;i<320*240;i++)
{
  if (we_want_to_display(depth[i]))
  {
    draw_on_image_at(colorPoints[i].X,colorPoints[i].Y);
  }  
}

That's the basics. If you look at the greenscreen example in the Kinect Developer Toolkit 1.5 it shows a good use for this.

Answer 2

This is a very common problem, something inherent in the math involved to sync the two images since the two cameras are sitting in two different places. Kind of like taking the two video feeds produced by a 3D camera and trying to sync them. They will always be slightly off.

Two ideas for correction:

1. Manually shift the bits from the depth image as you calculate it.
2. Add a shaker motor to the Kinect to reduce noise: http://www.youtube.com/watch?v=CSBDY0RuhS4 (I have found a simple pager motor to be effective.)

The MapDepthToColorImagePoint is for use in the Skeletal API to map a skeletal point to a depth point and then to an image point so that you can show joints on top of the RGB image.

Hope this helps.