Is it possible use keyboard actions without using the "PyAutoGUI" library on jenkins?

Question

I am using Robotframework to create test scripts for an application. This application requires many keyboard click/action combinations that are unavoidable. Right now I am using the PyAutoGui library to simulate these actions and they are working fine, but when I run them through a headless browser on Jenkins, those actions are not registered.

The error that I get is "PyAutoGUI fail-safe triggered from mouse moving to upper-left corner. To disable this fail-safe, set pyautogui.FAILSAFE to False." However, even after changing the Failsafe value to false, the keyboard action is still not captured.

The weird thing is that if someone is physically logged into the Jenkins box while the tests are running, the library works perfectly fine, but when running headless, the library breaks.

Is there another library I can possible use or a possible work around for this situation?

Thanks in advance!

Answer 1

The reason for such behaviour is that when there is no user logged in (physically or via RDP), there is no active desktop (think about all GUI elements, profiles, etc.) We had such issues in our environment. Here is a working solution:

1. create a job which will establish an RDP session on Windows VMs so there will be an active desktop. The job cannot end the RDP session. It needs to be run in background (usually no problem, as if somebody else logs in with the same user, the session will switch to the new user but the desktop will be active. )
2. make sure that, whenever you run your test jobs on Win VMs, there is already an RDP opened. Schedule job 1 to be run before tests on Win VMs.

From the technical details of job 1, we have our WinVM nodes named with the WIN prefix, so to get all Windows nodes, we query Jenkins via the Jenkins API.
Once we have a list of WinVMs (IP or hostnames), we run the following command on the Linux node looping with all discovered WinVMs nodes.

Basic command for one node :

   BUILD_ID=dontKillMe vncserver -kill :100 || true
   BUILD_ID=dontKillMe rm -rf /tmp/.X11-unix/X100 || true 
   BUILD_ID=dontKillMe vncserver :100
   BUILD_ID=dontKillMe DISPLAY=localhost:100
   BUILD_ID=dontKillMe export DISPLAY
   yum install -y freerdp
   ## loop through WinVMs below:    
   nohup xfreerdp -g <resolution> -u <user> -p <pas> <IP/hostname>
   ## end of loop

The magic is with nohup, as it runs the RDP session in the background after the job has been finished.

This is Centos with vncserver and xfreerdp installed.

#edit

You can ask the admin to create a WinVM for running tests, separating Jenkins with dev/test environment. In such way you could the open an RDP session on the node from anywhere or from Jenkins itself. For stability and performance, it is considered as a good practice not to run anything on the master.