iOS Automated Tests - XCTest vs Appium

Question

I am looking for opinions from test engineers and iOS developers regarding a specific matter.

I have been using Appium for over a year now, and I have come across various difficulties while running mobile web tests on real devices. Every time Apple makes changes in XCTest or Safari, Appium is directly impacted (even if they fix something, some other issue pops up in its place), and it has come to a point where I'm spending more time resolving Appium related issues as opposed to the ones from an AUT.

Furthermore, you would need an Apple computer to develop and execute Appium iOS tests anyway. So I thought why not use XCTest directly to write tests using Swift or whatever language that XCTest supports?

I would very much appreciate comments and opinions from people who have experience with both Appium and XCTest (Swift, etc) automated tests.

What are the advantages of using Appium to automate iOS native and web apps tests over XCTest?

Answer 1

As you've already discovered, one of the greatest disadvantages of Appium is that it's a third party framework and it breaks pretty much every time Apple makes an Xcode release. This is unlikely to change since Apple now maintains its own UI testing framework.

There are some other advantages to using Appium - you can pick from quite a few languages (Java, Ruby, Python, C#...), which makes it accessible to anyone who can program in one of those very popular languages, and you can conceivably share code between tests for the same app on iOS and Android.

In my experience, while there are a lot of people out there using Appium, the level of community support doesn't make up for the disappointing level of maintenance; aside from the fragility of Appium's compatibility with Xcode, I've found that some key functions have remained unimplemented in some bindings, e.g. scrolling in Python.

With UI tests, reliability is the most important aspect of your framework. Without reliability, you can't trust the tests to flag up problems, and without trust, your tests provide little to no value to you and your team. This is why I recommend XCTest over any third-party framework.

With XCTest, you never have to worry about not being able to update your version of Xcode, and the framework is maintained to Apple's release standards. As with all the iOS UI testing frameworks, there are some bugs, (particularly around pickers) but I find that the stability of the framework and the fact that it's owned by Apple outweighs the disadvantages of the odd bug.

Being endorsed by Apple is a significant pro for using XCTest, since Apple could remove access to the APIs which Appium depends on and Appium could stop working forever overnight. Historically, Apple do not simply remove support for their own frameworks without at least a year of notice.

To use XCTest directly, you need to use Swift (recommended) or Objective-C. There isn't as much language choice as Appium gives, but support for both languages is consistent as they both use the same implementation. Swift is a strong choice of language, especially for larger projects, because its type-safety allows you to notice many programming errors before runtime. Both languages also give you great intellisense (autocomplete) support in Xcode, which is something that is not offered out of the box by 'dynamic' languages like Python or Ruby.

The community around XCTest is growing as more information becomes available about using it with UI tests and more people feel able to adopt it. Many parts of the framework used for UI testing have been being used for many years for unit tests, so in many ways, there was already a lot of information available about using it, before the UI testing support was added.

Both frameworks use similar concepts - XCUIApplication is similar to Appium's Driver, which gives you access to what's on the screen. The level of functionality offered by both frameworks is arguably very similar, so it depends where your priorities lie - with reliability (XCTest) or reusability across other platforms and language accessibility (Appium).