Kinect pattern recognition [closed]

Question

I bought a Kinect to play around with on my PC in C#. (And what fun it is to program!)

I'm using CLNUIDevice.dll to get the depth image from Kinect. This works fine and I'm able to read the depth through the RGB values in the image.

Now I'm wondering what libraries there are to do different types of recognition. I've seen hand recognition, face recognition, skeleton, emotions, objects, etc.

The image processing libraries don't have to be for Kinect (though that would be nice) and they don't have to be for .Net (though that would be nice). For instance: Any object tracking lib will work, but if it can use the 3D heatmap from Kinect properly it will work much better.

Answer 1

You can take the series of RGB matrices produced by the Kinect and run them through standard image processing algorithms, in practice image processing algorithms are normally combined together to produce meaningful results. Here are a few standard techniques that could easily be implemented ( and combined) in .net:

Template Matching - a technique in digital image processing for finding small parts of an image which match a template image http://en.wikipedia.org/wiki/Template_matching

Morphological Image Processing - a theory and technique for the analysis and processing of geometrical structures, based on set theory, lattice theory, topology, and random functions http://ashleyaberneithy.wordpress.com/2011/08/08/automating-radiology-detecting-lung-nodules-using-morphological-image-processing-in-f/

There are also more advanced image processing techniques that can be used in specific scenarios, e.g face recognition and pattern matching via machine learning

Principle Component Analysis - I've used this technique in the past, and I think that this is used in modern consumer cameras to perform facial recognition http://en.wikipedia.org/wiki/Principal_component_analysis

Machine Learning pattern matching - I've used Support Vector Machines and Neural Network based learning algorithms in the past to detect patterns in image matrices. It's worth reading Vapnik's Statistical Learning Theory - http://www.amazon.com/Statistical-Learning-Theory-Vladimir-Vapnik/dp/0471030031 which shows how to successfully map training data into an n-dimensional structure, and how to successfully model hyperplanes within the structure which classify data, new data can then be classified based on this model. A library called LibSVM also exists which I have found useful. http://www.csie.ntu.edu.tw/~cjlin/libsvm/

Just a side note, it would probably be more natural to use F# within the .net world to implement some of these algorithms

EDIT : another really good book is "Digital Image Processing"