How I can attach the mouse movement (pyautogui) to pyvirtualdisplay with selenium webdriver (python)?

Question

I am trying to automatize a website how have a SWF inside.

I cant move the mouse with selenium, because is a SWF,so to fix this I use the pyautogui library.

Everything works fine!, but! when I use pyvirtualdisplay to hide the navigator the mouse is not attached, so I still see how pyautogui move my mouse.

My example code:

from selenium import webdriver
from pyvirtualdisplay import Display
import pyautogui

display = Display(visible=1, size=(1600,900))
display.start()


driver = webdriver.Firefox()
driver.set_window_size(1600,900)
driver.get('https://website.where.I.have.the.SWF.com')

sleep(5)
pyautogui.click(450, 180)

driver.close()
display.stop()

How I can attach the mouse to the pyvirtualdisplay instance?

Answer 1

You can monkey-patch pyautogui internals. Tested on 'xvfb' backend.

import os
from pyvirtualdisplay import Display
import pyautogui
import Xlib.display

v_display = Display(visible=1, size=(1600,900))
v_display.start()  # this changes the DISPLAY environment variable
# sadly, pyautogui does not detect this change
pyautogui._pyautogui_x11._display = Xlib.display.Display(
                os.environ['DISPLAY']
            )
...
pyautogui.click(...)  # clicks on v_display
...

v_display.stop()

Note: this should be sufficient to enable pyautogui mouse, using keyboard may require additional configuration of key mapping. For more info, please see: https://github.com/asweigart/pyautogui/blob/master/pyautogui/_pyautogui_x11.py