Algorithm to project 2D/3D objects onto a plane (Augmented Reality)?

Question

I'm working on a project about Augmented Reality using a marker (which is a chessboard) that takes images from webcam and insert a 2D/3D object onto the chessboard plane(realtime) .

By using openCV i was able to detect the chessboard corners and draw it.

I used Direct3D to render the frame (by using orthogonal projection), so that later I can use it to project 2d/3d objects.

Now I stuck at this stage. I know there are libraries such as ARToolkit etc, however I want to understand the technique and algorithm behind it. How can I find the transformation matrix that can map 3D object to the chessboard plane ?

Answer 1

This paper explains it well:

H. Kato and M. Billinghurst. Marker tracking and hmd calibration for a videobased augmented reality conferencing system. Augmented Reality, 1999. (IWAR ’99) Proceedings. 2nd IEEE and ACM International Workshop on, pages 85–94, 1999. 1, 2, 7, 9

A fast explanation:

1. Find corders of the marker and create 2d edges from it

2. create base vectors u1, u2 of the marker using planes defined by the edges (http://twitpic.com/4fmbsu)

3. use cross product on the two base vectors u1, u2 to obtain the third vector u3

4. use them to construct your matrix