Move to searched text on active screen with pyautogui

Question

I am trying to make a program that searches for a text on a web-page, then places the mouse cursor on the highlighted text after it has been found. Is this possible using pyautogui? If so, how. If not, are there any other alternatives to do this?

Example code below:

import webbrowser
import pyautogui


var = 'Filtered Questions'
webbrowser.open('https://stackexchange.com/')
time.sleep(2)

pyautogui.hotkey('ctrl', 'f')
pyautogui.typewrite(var)
#code to place mouse cursor to the occurrence of var

I would prefer to not use the pyautogui.moveTo() or pyautogui.moveRel() because the text I am searching for on the website is not static. The position of the searched text varies when the web page loads. Any help would be highly appreciated.

Answer 1

Yes, you can do that, but you additionally need Tesseract (and the Python-module pytesseract) for text recognition and PIL for taking screenshots.

Then perform the following steps:

1. Open the page
2. Open and perform the search (ctrl+f with pyautogui) - the view changes to the first result
3. Take a screenshot (with PIL)
4. Convert the image to text and data (with Tesseract) and find the text and the position
5. Use pyautogui to move the mouse and click on it

Here is the needed code for getting the image and the related data:

import time
from PIL import ImageGrab  # screenshot

import pytesseract
from pytesseract import Output
pytesseract.pytesseract.tesseract_cmd = (r"C:\...\AppData\Local\Programs\Tesseract-OCR\tesseract") # needed for Windows as OS

screen =  ImageGrab.grab()  # screenshot
cap = screen.convert('L')   # make grayscale

data=pytesseract.image_to_boxes(cap,output_type=Output.DICT)

print(data)

In data you find all required information you need to move the mouse and click on the text.

The downside of this approach is the ressource consuming OCR part which takes a few seconds on slower machines.