How to find a template in an image using a mask (or transparency) with OpenCV and Python?

Question

Let us assume we are looking for this template:

The corners of our template are transparent, so the background will vary, like so:

Assuming we could use the following mask with our template:

It would be very easy to find it.

What I have tried:

I have tried matchTemplate but it doesn't support masks (as far as I know), and using the alpha channel (transparency) in the template does not achieve this, as it compares the alpha channels instead of ignoring those pixels.

I have also looked into "region of interest", which I thought would be the solution, but with it you can only specify a rectangular area. I'm not even sure if it works on the template or not.

I'm sure this is possible to do by writing my own algorithm, but I was hoping this is possible via. standard OpenCV to avoid reinventing the wheel. Not to mention, it would most likely be more optimised than my own.

So, how could I do something like this with OpenCV + Python?

Answer 1

This could be achieved using only matchTemplate function, but a little workaround is needed.

Lets analyse the default metrics(CV_TM_SQDIFF_NORMED). According to matchTemplate documentation the default metrics looks like this

R(x, y) = sum (I(x+x', y+y') - T(x', y'))^2

Where I is image matrix, T is template, R is result matrix. Summation is done over template coordinates x' and y',

So, lets alter this metrics by inserting weight matrix W, which has the same dimensions as T.

Q(x, y) = sum W(x', y')*(I(x+x', y+y') - T(x', y'))^2

In this case, by setting W(x', y') = 0 you can actually make pixel be ignored. So, how to make such metrics? With simple math:

Q(x, y) = sum W(x', y')*(I(x+x', y+y') - T(x', y'))^2         = sum W(x', y')*(I(x+x', y+y')^2 - 2*I(x+x', y+y')*T(x', y') + T(x', y')^2)         = sum {W(x', y')*I(x+x', y+y')^2} - sum{W(x', y')*2*I(x+x', y+y')*T(x', y')} + sum{W(x', y')*T(x', y')^2)} 

So, we divided Q metrics into tree separate sums. And all those sums could be calculated with matchTemplate function (using CV_TM_CCORR method). Namely

sum {W(x', y')*I(x+x', y+y')^2} = matchTemplate(I^2, W, method=2) sum{W(x', y')*2*I(x+x', y+y')*T(x', y')} = matchTemplate(I, 2*W*T, method=2) sum{W(x', y')*T(x', y')^2)} = matchTemplate(T^2, W, method=2) = sum(W*T^2) 

The last element is a constant, so, for minimisation it does not have any effect. On the other hand, it still might me useful to see if our template have perfect match (if Q is approaching to zero). Nonetheless, for last element we actually do not need matchTemplate function, since it could be calculated directly.

The final pseudocode looks like this:

result = matchTemplate(I^2, W, method=2) - matchTemplate(I, 2*W*T, method=2) + as.scalar(sum(W*T^2)) 

Does it really do exactly as defined? Mathematically yes. Practically, there is some small rounding error, because matchTemplate function works on 32-bit floating-point, but I believe it is not a big problem.

Please note, that you can extent analysis and have weighted equivalents for any metrics offered by matchTemplate.

This actually worked for me. I am sorry I don't give actual code. I am working in R, so I don't have the code in Python. But idea is quite straightforward.

I hope this will help.

Answer 2

What worked for me the one time I needed this was to fill the "mask" areas with white noise. Then it gets effectively washed out of the correlation when looking for matches. Otherwise I got, as I presume you did, false matches on the masked areas.