python - How to recognize images and click on them

Question

I would like to make a script that clicks on images depending on what is asked, it needs to go trough a list of images. so for example if the user is asked by the program to click on the green circle:

question_list = greencircle, redcircle, bluesquare, redtriangle

if(greencircle == greencircle.png){
    pyautogui.click(greencircle.png)
}

could someone help with this?

Answer 1

PyAutoGUI has a built in function called locateOnScreen() which returns the x, y coordinates of the center of the image if it can find it on the current screen (it takes a screenshot and then analyzes it).

The image has to match exactly for this to work; i.e. if you want to click on a button.png that button picture has to be the same exact size / resolution as the button in your windows for the program to recognize it. One way to achieve this is to take a screenshot, open it in paint and cut out only the button you want pressed (or you could have PyAutoGUI do it for you as I'll show in a later example).

import pyautogui

question_list = ['greencircle', 'redcircle', 'bluesquare', 'redtriangle']

user_input = input('Where should I click? ')

while user_input not in question_list:
    print('Incorrect input, available options: greencircle, redcircle, bluesquare, redtriangle')
    user_input = input('Where should I click?')

location = pyautogui.locateOnScreen(user_input + '.png')
pyautogui.click(location)

The above example requires you to already have greencircle.png and all the other .png in your directory

PyAutoGUI can also take screenshots and you can specify which region of the screen to take the shot pyautogui.screenshot(region=(0, 0, 0, 0)) The first two values are the x,y coordinates for the top left of the region you want to select, the third is how far to the right(x) and the fourth is how far down (y).

This following example takes a screenshot of the Windows 10 Logo, saves it to a file, and then clicks on the logo by using the specified .png file

import pyautogui

pyautogui.screenshot('win10_logo.png', region=(0, 1041, 50, 39))
location = pyautogui.locateOnScreen('win10_logo.png')
pyautogui.click(location)

You also don't have to save the screenshot to a file, you can just save it as a variable

import pyautogui

win10 = pyautogui.screenshot(region=(0, 1041, 50, 39))
location = pyautogui.locateOnScreen(win10)
pyautogui.click(location)

Making a program detect if a user has clicked in a certain area (let's say, the windows 10 logo) would require another library like pynput.

from pynput.mouse import Listener    

def on_click(x, y, button, pressed):
    if 0 < x < 50 and 1080 > y > 1041 and str(button) == 'Button.left' and pressed:
        print('You clicked on Windows 10 Logo')
        return False    # get rid of return statement if you want a continuous loop

with Listener(on_click=on_click) as listener:
    listener.join()

PUTTING IT ALL TOGETHER

import pyautogui
from pynput.mouse import Listener

win10 = pyautogui.screenshot(region=(0, 1041, 50, 39))
location = pyautogui.locateOnScreen(win10)

# location[0] is the top left x coord
# location[1] is the top left y coord
# location[2] is the distance from left x coord to right x coord
# location[3] is the distance from top y coord to bottom y coord

x_boundary_left = location[0]
y_boundary_top = location[1]
x_boundary_right = location[0] + location[2]
y_boundary_bottom = location[1] + location[3]


def on_click(x, y, button, pressed):
    if x_boundary_left < x < x_boundary_right and y_boundary_bottom > y > y_boundary_top and str(button) == 'Button.left' and pressed:
        print('You clicked on Windows 10 Logo')
        return False    # get rid of return statement if you want a continuous loop


with Listener(on_click=on_click) as listener:
    listener.join()