OpenCV template matching, multiple templates

Question

Im trying to make a bot for a game. Basically it picks up items from the ground, thing is these items look different sometimes for ex. the angle is different or they are lying on differentyl coloured ground etc. To make everything work I need multiple templates. Is there any way to do that? If you dont understand just tell me in the comments. Here is what I tried so far:

files = ["bones_{}.png".format(x) for x in range(6)]

    for i in range(6):
        img_gray = cv2.cvtColor(imageGrab(), cv2.COLOR_BGR2GRAY)
        f = str(files[i])
        template = cv2.imread(f, 0)
        w, h = template.shape[:: -1]
        res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
        threshhold = 0.70
        loc = np.where( res >= threshhold)

This works but It could be better. Do you have any ideas?

Answer 1

In your current code there are a lot of steps performed multiple times where once (per frame) suffices. You can gain efficiency by separating those out.

You're currently reloading the templates on every frame, this is very inefficient as you'll easily get 100+ loads per second. Instead create a list that holds the templates so they remain in memory. Accessing from memory is much faster than loading from disk.
You can do the same for the width/length of the templates, but it is not actually used in your code, so maybe you can skip it all together.
The threshold needs to be set only once.

templates = []
templ_shapes = []
threshold = 0.70

for i in range(6):
    templates.append(cv2.imread("bones_{}.png".format(i),0))
    templ_shapes.append(templates[i].shape[:: -1])

All the templates can be compared to the same screengrab, so you should take that outside the for loop. This is an easy but pretty big win. So on every frame, grab the screen once, and match all templates. For clarity and convenience you can put it in a function:

def doTemplateMatch():
    img_gray = cv2.cvtColor(imageGrab(), cv2.COLOR_BGR2GRAY)
    for template in templates: 
        res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
        loc = np.where( res >= threshold)