Project image onto notebook using OpenCV

Question

I am trying to implement an application that projects an image onto a page of a notebook, using OpenCV, a webcam and a projector. To achieve that, I am doing the following steps:

1. I am using a webcam to detect the four corners points of a page.
2. A homography is learned between the four corner points of the camera image and their projections on my desk, as seen in the camera. By using the inverse transformation, I will be able to know where I should draw something on my camera image, so that the projection "ends up" at a desired location.
3. I am applying the inverse transformation to the detected four corners points of the page.
4. I am warping the desired image to the new, transformed set of points.

So far it works well, if the notebook is on my desk and wide open. Like in this picture:

But if I try to close one side (or both), the following happens:

See the problem? In the first picture the image is perfectly aligned with the edges of the page and remains so if you rotate or translate the notebook, while keeping it on the desk. But that doesn't happen in the second image, where the the top edge of the image is no longer parallel to the top edge of the page (the image becomes more and more skewed).

Can anyone explain why I get this projection problem or at least point me to some resources where I can read about it? I need to mention that the projector and the webcam are placed above and to the left of the notebook, not right above them.

Any tips or suggestions are welcome. Thank you!

Answer 1

You want an effect that is called a key stone correction. The problem you are experiencing is most probably due to the fact that optical axes, positions, and focal lengths of a web camera and a projector are different. I suggest to calibrate your setup so you would know their relative pose and incorporate it in your inverse Homography.