Headless automation of IE-browser, tracking site rendering times

Question

I'm need to monitor my sites render times for common tasks (Login, Search etc.). I need something automated that can mimic a users actions on I.E. and be able time how long a page takes to render.
Example automated execution:

1) open headless IE browser

2) go to http://google.com

3) type "stackoverflow"

4) press submit button

5) start timer

6) wait for results page to fully render

7) stop timer

8) Close IE

9) record results

I need this to run as a scheduled task while the server, without the user logged in.

I have been searching for something to help me do so. Anyone have any experience with this type of thing or know of anything that can accomplish this?

Answer 1

It depends on what you focus, functionality or performance.

Functionality

When monitoring functionality, you aim at automatically ensuring that a web application still works correctly. Usually, this is more part of the continous integration process - and less part of production monitoring. It can be well done with HtmlUnit, Selenium or WebDriver. HttpUnit is no longer recommended (API more low-level, JavaScript not so well supported, less widely adopted, fewer bug fixes and enhancements).

HtmlUnit simulates a browser. So you can never be sure, that your application behaves exactly identical in a real browser. This is especially important for sophisticated Ajax applications. This is comparable to all the small incompatibilities between FireFox and Internet Explorer. Pros: Headless, easy to understand. Cons: Risk of undetected incompatibilities.

Selenium remote controls a real browser. In our setup, we could not use it headlessly, especially with Internet Explorer. But if you embed it into a virtual machine, it runs headlessly. If your application is reachable through public internet, you might even use Selenium Grid and a preconfigured virtual machine from the Amazon Elastic Cloud EC2. Pros of Selenium: Real world compatibility, easy scripting. Cons: Headless only in virtual machine, performance overhead, more complex runtime setup, stress simulation of concurrent users only in the cloud.

Up to version 1.5, Selenium uses a JavaScript part called Selenium Core to control the browser. If your application has security restrictions for JavaScript, Selenium might not work correctly.

WebDriver uses for each browser the specific interface, e.g. for FireFox an extension and for internet explorer Automation Controls. Additionally it uses the operation system, e.g. for simulating keystrokes. This is more powerful, robust and reliable than Selenium Core. As of Selenium version 2.0, WebDriver is integrated into Selenium. But Selenium 2.0 is still beta.

Performance

You mention measuring with a timer and you mention rendering times. When monitoring performance of a web application, you want to be alerted when real world usage of a application is no longer possible due to overlong answering times.

In this scenario, you are normally not interested in exact results in milliseconds. You can still use one of the tools mentioned above. For example, a browser with Selenium Core is slower than a real world browser - but this is of little relevance for continuous monitoring.

If you absolutely need exact measurements, none of the above is suitable. You should differentiate between client-side duration and network plus server-side duration.

• Client-side duration is needed for rendering the HTML and for executing the JavaScript. It does not depend on the number of concurrent users. You can measure it once, e.g. with Firebug. You do not need to monitor it permanently.

• Network plus server-side duration is needed for transferring the request to the server, handling the request and generating the response and transferring the response to the client. They vary according to network usage and number of concurrent users. You can exactly measure and monitor them for example with JMeter. But in case of sophisticated Ajax functionality, the simulation of the right client requests in JMeter is a complex task. Pros of JMeter: Exact measurement, possibility to stress an application with many concurrent users. Cons: Limited for Ajax, much effort for request building.

Answer 2

Another option might be Selenium Remote Control (or Selenium in general).