Feature matching with flann in opencv

Question

I am working on an image search project for which i have defined/extracted the key point features using my own algorithm. Initially i extracted only single feature and tried to match using cv2.FlannBasedMatcher() and it worked fine which i have implemented as below:

Here vec is 2-d list of float values of shape (10, )
Ex:
[[0.80000000000000004, 0.69999999999999996, 0.59999999999999998, 0.44444444444444448, 0.25, 0.0, 0.5, 2.0, 0, 2.9999999999999996]

[2.25, 2.666666666666667, 3.4999999999999996, 0, 2.5, 1.0, 0.5, 0.37499999999999994, 0.20000000000000001, 0.10000000000000001]

[2.25, 2.666666666666667, 3.4999999999999996, 0, 2.5, 1.0, 0.5, 0.37499999999999994, 0.20000000000000001, 0.10000000000000001]

[2.25, 2.666666666666667, 3.4999999999999996, 0, 2.5, 1.0, 0.5, 0.37499999999999994, 0.20000000000000001, 0.10000000000000001]]

vec1 = extractFeature(img1)
vec2 = extractFeature(img2)

q1 = np.asarray(vec1, dtype=np.float32)
q2 = np.asarray(vec2, dtype=np.float32)

FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks=50)   # or pass empty dictionary
flann = cv2.FlannBasedMatcher(index_params,search_params)
matches = flann.knnMatch(q1,q2,k=2)

But now i have one more feature descriptor for each key point along with previous one but of different length. So now my feature descriptor has shape like this:

[[[0.80000000000000004, 0.69999999999999996, 0.59999999999999998, 0.44444444444444448, 0.25, 0.0, 0.5, 2.0, 0, 2.9999999999999996],[2.06471330e-01,   1.59191645e-02,   9.17678759e-05, 1.32570314e-05,   4.58424252e-10,   1.66717250e-06,6.04810165e-11]

[[2.25, 2.666666666666667, 3.4999999999999996, 0, 2.5, 1.0, 0.5, 0.37499999999999994, 0.20000000000000001, 0.10000000000000001],[ 2.06471330e-01,   1.59191645e-02,   9.17678759e-05, 1.32570314e-05,   4.58424252e-10,   1.66717250e-06, 6.04810165e-11],

[[2.25, 2.666666666666667, 3.4999999999999996, 0, 2.5, 1.0, 0.5, 0.37499999999999994, 0.20000000000000001, 0.10000000000000001],[ 2.06471330e-01,   1.59191645e-02,   9.17678759e-05, 1.32570314e-05,   4.58424252e-10,   1.66717250e-06, 6.04810165e-11],

[[2.25, 2.666666666666667, 3.4999999999999996, 0, 2.5, 1.0, 0.5, 0.37499999999999994, 0.20000000000000001, 0.10000000000000001],[ 2.06471330e-01,   1.59191645e-02,   9.17678759e-05, 1.32570314e-05,  4.58424252e-10,   1.66717250e-06, 6.04810165e-11]]

Now since each point's feature descriptor is a list two lists(descriptors) with different length that is (10, 7, ) so in this case i am getting error:

setting an array element with a sequence.

while converting feature descriptor to numpy array of float datatype:

q1 = np.asarray(vec1, dtype=np.float32)

I understand the reason of this error is different length of lists, so i wonder What would be the right way to implement the same?

Answer 1

You should define a single descriptor of size 10+7=17.

This way, the space descriptor is now of 17 and you should be able to use cv2.FlannBasedMatcher.

Either create a global descriptor of the correct size desc_glob = np.zeros((nb_pts,17)) and fill it manually or find a Python way to do it. Maybe np.reshape((nb_pts,17))?

Edit:

To not favor one descriptor type over the other, you need to weight or normalize the descriptors. This is the same principle than computing a global descriptor distance from two descriptors:

dist(desc1, desc2) = dist(desc1a, desc2a) + lambda * dist(desc1b, desc2b)