mercurial: running remote regression tests automatically on every commit

Question

I commit every time I make some changes that I think might work: I don't do extensive testing before a commit. Also, my commits will soon be automatically pushed to a remote repository. (I'm the only developer, and I have to add features or rewrite parts of the code many times a day.)

I'd like to set up a remote computer to run regression tests automatically whenever I commit anything; and then email me back the differences report.

What's the easiest way to set this up?

All my code is in Python 3. My own system is Windows 7, ActiveState Python, TortoiseHG, and Wing IDE. I can set up the remote computer as either Linux or Windows. The application is all command-line, with text input and output.

Answer 1

Use a continious integration server such as Buildbot or Jenkins and configure it to monitor the repository. Then run the tests using that. Buildbot is written in Python so you should feel right at home with it.

If you feel it's wasteful to make Buildbot or Jenkins poll the repository (even though hg pull uses very few resources when there are no new changesets), then you can configure a changegroup hook in the repository to trigger a build in the CI server.

Answer 2

I would recommend setting up Buildbot. You can have it watch a remote repository (Mercurial is supported) and automatically kick off a build when the repository changes. In your case, a build would just be running your test suite.

Its waterfall display allows you to see which builds failed and when, in relation to commits from the repository. It can even notify you, with the offending commit, when something breaks.

Jenkins is another option, supporting most of the same features. There are even cloud hosting options, like ShiningPanda that can host it for you, and they offer free licensing for open-source projects.