Find world space coordinate for pixel in OpenCV

Question

I need to find the world coordinate of a pixel using OpenCV. So when I take pixel (0,0) in my image (that's the upper-left corner), I want to know to what 3D world space coordinate this pixel corresponds to on my image plane. I know that a single pixel corresponds to a line of 3D points in world space, but I want specific the one that lies on the image plane itself.

This is the formula of the OpenCV Pinhole model of which I have the first (intrinsics) and second (extrinsics) matrices. I know that I have u and v, but I don't know how to get from this u and v to the correct X, Y and Z coordinate.

What I've tried already:

• I thought to just set s to 1 and make a homogeneous coordinate from [u v 1]^T by adding a 1, like so: [u v 1 1]^T. Then I multiplied the intrinsics with the extrinsics and made it into a 4x4 matrix by adding the following row: [0 0 0 1]. This was then inverted and multiplied with [u v 1 1]^T to get my X, Y and Z. But when I checked if four pixels calculated like that lay on the same plane (the image plane), this was wrong.

So, any ideas?

Answer 1

IIUC you want the intersection I with the image plane of the ray that back-projects a given pixel P from the camera center.

Let's define the coordinate systems first. The usual OpenCV convention is as follows:

• Image coordinates: origin at the top-left corner, u axis going right (increasing column) and v axis going down.
• Camera coordinates: origin at the camera center C, z axis going toward the scene, x axis going right and y axis going downward.

Then the image plane in camera frame is z=fx, where fx is the focal length measured in pixels, and a pixel (u, v) has camera coordinates (u - cx, v - cy, fx).

Multiply them by the inverse of the (intrinsic) camera matrix K you'll get the same point in metrical camera coordinates.

Finally, multiply that by the inverse of the world-to-camera coordinate transform [R | t] and you'll get the same point in world coordinates.