Which openCv function can be used to compute BEV perspective transformation given a point coordinates and the camera extrinsics/intrinsics?

Question

I have the 3x3 intrinsics and 4x3 extrinsics matrices for my camera obtained via cv2.calibrateCamera()

Now I want to use these paramenters to compute the BEV (Bird Eye View) transformation for any given coordinates in a frame obtained from the camera.

Which openCv function can be used to compute the BEV perspective transformation for given point coordinates and the camera extrinsics and/or intrinsics 3x3 matrices?

I found something very related in the following post: https://deepnote.com/article/social-distancing-detector/ based on https://www.pyimagesearch.com/2014/08/25/4-point-opencv-getperspective-transform-example/ ,

they are using cv2.getPerspectiveTransform() to get a 3X3 matrix, but I don't know whether this matrix represents the intrinsics, the extrinsecs or something else. Then they are transforming the list of points using such matrix in the following way:

#Assuming list_downoids is the list of points to be transformed and matrix is the one obtained above
list_points_to_detect = np.float32(list_downoids).reshape(-1, 1, 2)
transformed_points = cv2.perspectiveTransform(list_points_to_detect, matrix)

I really need to know if I can use this cv2.perspectiveTransform function to compute the transformation or if there's another better way to do this using the extrinsics, the intrinsics or both, without having to reuse the frame, since I already have the detected/selected coordinates saved in an array.

Answer 1

The answer is : it is impossible to compute a BEV of a scene if you do not have distance-related information about the pixels of your image.

Think about it : imagine you have the picture of a vertical screen : the Bird's Eye View would then be a line. Now say that this screen is displaying the image of a landscape and that the picture of this screen is indistinguishable from a picture of the landscape itself. The BEV would still be a line (a colorful one though).

Now, imagine you have the exactly the same picture, but this time it's not a picture of a screen but of the landscape. Then, the Bird's Eye View is not a line and is closer to what we usually imagine a BEV to be.

Finally, let me state that OpenCV has no way to know if your picture is describing a plane of something else (even given camera parameters), therefore, it cannot compute the BEV of your scene. The function cv2.perspectiveTransform needs you to pass it a homography matrix (you may obtain one using cv2.findHomography(), but you will need some distance information about your image as well).

Sorry about the negative answer, but there's no way to solve your problem given only the intrinsic and extrinsic calibration matrices of the camera.