Sikuli, selenium testing on Jenkins: Allow the browser to be launched in the foreground, as it is when i run it from development machine?

Question

The issue is that Sikuli's image recognition capabilities only work when the target of the Sikuli tests is in the foreground and it has full access to the mouse. In its current configuration, Jenkins projects are never visible from the desktop of the windows slave node.

The way my current automated testing suite is setup right now, about 30 tests are written using only selenium. In the final two tests, selenium starts a webdriver (currently chromedriver but can be firefox or IE) and navigates to the right pages and Sikuli does its thing from there. Both use the java language bindings.

Currently Sikuli appears to be installed correctly, but when it is asked to click through a list of screenRegions supposedly populated by matching the desktop with a target image, it immediately throws a NPE.

How do I change my Jenkins setup on my windows 7 slave to allow the browser to be launched in the foreground, as it is when i run it from my windows 7 development machine?

Answer 1

So you've got a development machine where Sikuli works, and a test machine where Jenkins runs Sikuli and it doesn't work.

I had this same setup (with the Sikuli tester/Jenkins slave running Windows 7). What worked for me was to have the test machine run a VNC server, log in as a the tester account over VNC, and launch the Jenkins slave as a regular application and not a service. I used UltraVNC server.

As far as I could tell, the problem stems from Windows' security measures, which are designed to prevent a remote user from controlling your machine without your presence.

If you run Jenkins as a service, it doesn't get a real desktop allocated to it. The script will run, but (as you're seeing) Sikuli won't actually be able to find anything because there's nothing for it to look at. (Selenium is inspecting the page content programmatically, so it doesn't mind that the page is not actually displayed on any screen.)

You can also just run Jenkins and leave yourself logged in, and skip the VNC server. I wanted to run the machine without a monitor or keyboard and still have control, though. If that's your goal as well, you can't use Remote Desktop, because it allocates a new desktop on connection and then destroys it when you disconnect. (So Jenkins will work when you're watching and fail when you leave for the night. Very frustrating!)

As a side benefit, if you do it this way you can connect over VNC and watch Jenkins as it's running the test.