WatiN or Selenium? [closed]

Question

Just want to say that I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.

So no concerns there.

Hope this helps in making your choice Jeroen van Menen Lead dev WatiN

If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:

As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.

I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.

We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.

I've been trying 'em both out and here are my initial thoughts...

---

WatiN

The Good

• Fast execution.
• Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
• Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
• NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.

The Bad

• Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
• What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
• No native XPath support.
• No out-of-the-box remote execution/grid based execution.

Script Example (C#). You can't do this with Selenium (not that I know off, at least):

class IEManager
{
    IE _ie = null;
    object _lock = new object();

    IE GetInstance(string UrlFragment)
    {
        lock (_lock)
        {
            if (_ie == null)
            {
                var instances = new IECollection(true);  //Find all existing IE instances
                var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
                _ie = match ?? new IE();
                if (match==null)  //we created a new instance, so we should clean it up when done!
                    _ie.AutoClose = true;
            }
        }

        return _ie;
    }
}

---

Selenium

• Slower than WatiN (especially since a new process has to be created).
• Built-in CSS selectors/XPath support.
• Selenium IDE is good (can't say great, but it’s the best in class!).
• Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
• Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
• Support chrome and other browsers out of the box.

Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.

The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.

Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.

In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.

Neither. Use Coypu. It wraps Selenium. Much more durable. https://github.com/featurist/coypu

Update Ye Oliver you're right. Ok why's it better? Personally I've found the Selenium driver for IE in particular to be very fragile - there's a number of 'standard' driver exceptions that I've time again found when driving Selenium for Unit Tests on ajax heavy websites.

Did I mention I want to write my scripts in c# as a Test Project ? Yes Acceptance Tests within a continous build deployment.

Well Coypu deals with the above. It's a wrapper for Selenium that allows test fixtures such as,

browser.Visit("file:///C:/users/adiel/localstuff.htm")
browser.Select("toyota").From("make");
browser.ClickButton("Search");

... which will spin up a (configurable brand of) browser and run the script. It works great with scoped regions and is VERY extendable.

There's more examples at GitHub and as Olvier below mentions, Adrian's video is excellent. I think it's the best way to drive browser based tests in the .Net world and tries to follow it's Ruby namesake capybara

I've used both, they both seem to work ok. My nod is for Selenium as it seemed to have better Ajax support. I believe WaTiN has matured though since last I used it so it should have the same thing.

The biggest thing would be which development environment do you like to be in? Selenium and Watin have recorders but Selenium is in the browser and watin is in visual studio. + and -'s to both of those.