Is Selenium testing worthwhile on mobile devices?

Question

We've been planning to do UI testing of our web application using Selenium. We're now having second thoughts. From what I can tell, the webdrivers for Android and iOS devices don't use the actual browser that a user would use. Instead they use a WebView, which may or may not be similar to the actual browser. I've seen various reports for Android in particular that seem to indicate that the WebView behaves differently than the built in browser and from Chrome, which is now the default on some devices.

Do you find that using Selenium to test on the actual hardware device is worthwhile? Is it more or less accurate than using emulators? What have you found is the most effective way to use automated testing to make you comfortable that your web application will work correctly on all of the hardware platforms that you support?

In our case, we need to support recent versions of major browsers on PCs, Macs, tablets, and phones. Selenium is working well for the PC testing, but I'm concerned about the tablet and phone testing. We're considering whether we should roll our own or use something like uitest.js that we can actually run on the actual device browsers.

Answer 1

While both the iPhone driver and Android driver that exist in the Selenium WebDriver source code repository today use a WebView control to browse the web, both projects have been largely deprecated. This deprecation is explicit in the case of the iPhone driver, and tacit in the case of the Android driver.

Those drivers have been supplanted by other projects like ios-driver, Selendroid, and Appium. These projects are based on the WebDriver project, using the same client API and communicate using the same JSON-over-HTTP-based wire protocol, but are capable of automating native and hybrid applications on mobile platforms. They can usually automate the application either on the device or in an emulator. In the case of each of the three projects I've mentioned, they are actively maintained and development is ongoing.

Answer 2

From my experience, and my personal opinions -

Testing on ALL browsers, be it Android, and webkit based browsers is not always necessary.

A starting point should be to look at what browsers actually hit your application. Take a look at some network graphs, and if Firefox or Chrome is the main traffic to your app, then you should cover those browsers first. All other browsers are just additives. Your primary responsibility should be to cater to those who use your app most.

In the mobile market - automating your mobile applications isn't always completely necessary as unit tests are usually sufficient because they point the same service points.

In my personal opinion - mobile testing is overrated anyway.. Why not just change the user agent of Chrome, or whatever browser you are using, to make your app think it's android or iOS?

Answer 3

Now Android driver is removed and replaced by Selendroid for Android.

1. Java 6+
2. Latest Android-Sdk must be installed and ANDROID_HOME set
3. Download selendroid.jar file

4. Launch Server by

   java -jar selendroid-standalone-0.17.0-with-dependencies.jar -app selendroid-test-app-0.17.0.apk