Understanding the view and projection matrix from pybullet

Question

When working with rendering images in Pybullet, one has to use getCameraImage which takes as inputs a view and projection matrices (pybullet also have functions to generate those matrices). In theory, the projection matrix should be P = K[R|t], it can be re-written as P = [M|-MC] so we could use in theory RQ decomposition with M where R is an upper triangular matrix. So we could recover K and [R|t] from the projection matrix (having in mind that the R from RQ decomposition is not the R from R|t). But when I use for example scipy.linalg.rq the result is not a valid K (intrinsic) matrix.

Can someone explain how is the projection matrix exactly defined and what is the view matrix in pybullet? and how we can retrieve the intrinsic and extrinsic parameters using those matrices?

Answer 1

So pybullet usually constructs the projection matrix (source code) using the field of view (FOV in rads) as

and the intrinsic matrix is defined as

p_x and p_y are the principal points, usually the centre of the image. So there are a few differences:

1. The dimensions. Pybullet adds the third row (not the forth) and the fourth column to keep depth information.
2. Ignoring the third row the element 2,2 (zero-index) is not 1.
3. Pybullet is using a 0 skewed parameter.
4. It is not using the focal length (it is but it is calculating it from the FOV).
5. Pybullet is assuming p_x = p_y = 0

First, pybullet uses the notation of OpenGL so it is using a major-column order (read more). Meaning the first element while indexing is the column and not the row. Therefore the actual projection matrix from pybullet should be transposed.

Second, the full equation to convert FOV to focal length f is:

therefore pybullet is multiplying the focal length by 2/h. The reason why is because pybullet uses Normalized Device Coordinates (NDC) which clips the value between [-1,1] (dividing x by width clips it to [0,1] and multiplying it by 2 clips it between [0,2] which if the principal point is in the middle point of the image 1,1 then it is clipped to [-1,1]). Therefore pybullet's focal length is the proper focal length using NDC.

The non-zero values in the third column of the projection matrix are there to map z values in OpenGL so we can ignore them.

k, l in the K matrix is the ratio mm/px, if we are using pybullet we can say k=l=1.

Some useful resource are [1], [2] and [3].