Strategy for Automated UI testing on remote virtual machines

Question

I'm using TeamCity for my CI builds, and I'd like to set up a second build for running automated UI tests on Windows XP and Windows 7 virtual machines.

I imagine the build working as follows:

1. Compile, run unit tests, etc.
2. Prepare MSI using WiX
3. Copy MSI to target test machines
4. Remotely execute MSI's
5. Copy test harness code to remote machine
6. Run tests
7. Build finishes

The automated UI tests are written using NUnit and would need to be run directly on the test virtual machine (they can't run remotely). It's important that if the tests fail, it appears in the TeamCity build log and the build fails. I'd rather not install VS or the TeamCity build agents on either of the test virtual machines.

It seems that most of this should be possible using psexec.exe. Are there any alternative (preferably open source) tools that I should look at?

Answer 1

takes a deep breath

We were looking into something to help us out with our automated UI tests. We use ranorex to test the UI and TeamCity/Msbuild to execute the tests.

We never found any tools to help us out (I’m constantly keeping an eye out for some so will monitor this thread) but here is what we did instead.

1. The CI server copies the setup files and test scripts to the Testing Host Server.
2. The CI server then launches a custom app on the Testing Host Server providing the name of the VM to launch.
3. The Test Host Server then launches the VM software, using Virtual PC.exe -singlepc -pc vhdname.vhd -launch , and waits for it to shutdown (after it’s run its tests).
4. The VM grabs the setup files and scripts from the network location and executes.
5. After the tests are run it then returns the results to a networked location and shuts itself down.
6. Control is returned to the custom app.
7. Control is returned to the CI server which determines from the results if it has passed or failed (and updates the UI so developers are made aware of the result).
8. Results are collection as artifacts in TeamCity and tagged in Svn.

I think that's everything. Its convoluted, however, it works. Hope someone of that helps you.

Answer 2

Jeff Brown of the Gallio team has been talking about a tool called Archimedes that he's planning to write to support this kind of requirement. It sounds promising, but I don't think there has been much progress on it so far.

In the mean time though, there is something in the Gallio project called VM Tool that may do what you want. It provides commands to stop, start and snapshot virtual machines, and more importantly, to copy files back and forth and execute commands.

I presume you have also considered Powershell Remoting?