OpenCV / Python Bundle adjustment

Question

I have an application where I am attempting structure from motion for multiple views from the same camera using OpenCV in Python. (OpenCV isn't a hard requirement but Python is right now). Example: I have 16 camera images to work with with small motions between each frame. In the frames (200x200) I can track ~50 features and I'd like to estimate camera pose and 3D point locations for the features as accurately as possible.

I'm at the point where I'd like to use a stream of frames (from video) to refine the estimates of camera pose and 3D point positions since estimating pose and triangulating 3D points from two frames produces a lot of noise. I believe Bundle Adjustment is the next obvious direction to go in, but I'm not finding any obvious Python implementation of bundle adjustment to use. Many options (such as LevMarqSparse::bundleAdjust()) appear to be only partially completed or not fully adopted.

What is a good place to start? I guess I'm looking for a relatively simple Python bundle adjustment to prototype with and see if that is a direction I want to invest more time into.

Answer 1

Assuming you have a calibrated camera and an initial estimate of intrinsics and extrinsic parameters, you could at first perform a simple bundle adjustment directly in Python. For instance, you could use stacks of three images, compute 3D points from the feature points via the homogeneous triangulation method. A simple bundle adjustment as a first prototype can be build for instance via scipy.least_squares and the Trusted Region Reflective non-linear optimization technique. Check out this tutorial.

Afterwards you can decide, whether you want to implement or use a Levenberg-Marquardt optimization technique which is able to handle sparse Jacobians or even determine the Jacobian analytically to hypothetically increase the convergence if desired.

So far, I think there is no adequate library in Python which provides a potent and high performant implementation of bundle adjustment.