Surface normal on depth image

Question

How to estimate the surface normal of point I(i,j) on a depth image (pixel value in mm) without using Point Cloud Library(PCL)? I've gone through (1), (2), and (3) but I'm looking for a simple estimation of surface normal on each pixel with C++ standard library or openCV.

Answer 1

You need to know the camera's intrinsic parameters, so that you can also know the distance between pixels in the same units (mm). This distance between pixels is obviously true for a certain distance from the camera (i.e. the value of the center pixel)

If the camera matrix is K which is typically something like:

    f  0  cx
K=  0  f  cy
    0  0   1

Then, taking a pixel coordinates (x,y), then a ray from the camera origin through the pixel (in camera world coordinate space) is defined using:

              x
P = inv(K) *  y
              1

Depending of whether the distance in your image is a projection on the Z axis, or just a euclidean distance from the center, you need to either normalize the vector P such that the magnitude is the distance to the pixel you want, or make sure the z component of P is this distance. For pixels around the center of the frame this should be close to identical.

If you do the same operation to nearby pixels (say, left and right) you get Pl and Pr in units of mm Then just find the norm of (Pl-Pr) which is twice the distance between adjacent pixels in mm.

Then, you calculate the gradient in X and Y

gx = (Pi+1,j - Pi-1,j) / (2*pixel_size)

Then, take the two gradients as direction vectors:

ax = atan(gx),  ay=atan(gy)


     | cos ax    0    sin ax |   |1|
dx = |    0      1       0   | * |0|
     | -sin ax   0    cos ax |   |0|

     |    1      0       0   |   |0|
dy = |    0   cos ay -sin ay | * |1|
     |    0   sin ay  cos ay |   |0|

N = cross(dx,dy);

You may need to see if the signs make sense, by looking at a certain gradient and seeing of the dx,dy point to the expected direction. You may need to use a negative for none/one/both angles and same for the N vector.