Building XCTest UI tests for AWS Device Farm

Question

I'm trying to set up our iOS XCUITests to run on AWS Device Farm, but it seems no matter how I build and upload them, the tests will not run. They execute and pass when run locally in XCode, but do not run on AWS Device Farm.

I've set up the absolute simplest app I can manage to try to isolate this issue. It consists of a single View with a single Label, and a UI Test which verifies that the label exists.

Here is the error that appears in AWS Device Farm's interface.

Digging into the log files reveals this (in the "Application Output" log file):

2016-04-03 19:07:39.842 XCTRunner[195:28521] Running tests...
2016-04-03 19:07:39.868 XCTRunner[195:28521] Unable to load configuration data from specified path ; error: The file name is invalid.
2016-04-03 19:07:39.872 XCTRunner[195:28521] Looking for test bundles in /var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns
2016-04-03 19:07:39.875 XCTRunner[195:28521] Found test bundle at /var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns/MyTestAppUITests.xctest
2016-04-03 19:07:39.878 XCTRunner[195:28521] Looking for configurations in /var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns/MyTestAppUITests.xctest
2016-04-03 19:07:39.879 XCTRunner[195:28521] No configurations found, creating a default configuration that will run all tests.
2016-04-03 19:07:39.993 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
2016-04-03 19:07:39.995 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
2016-04-03 19:07:39.995 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
Test Suite 'Selected tests' started at 2016-04-03 19:07:40.008
Test Suite 'MyTestAppUITests' started at 2016-04-03 19:07:40.014
Test Case '-[MyTestAppUITests testExample]' started.
2016-04-03 19:07:40.021 XCTRunner[195:28521] *** Assertion failure in -[XCUIApplication init], /Library/Caches/com.apple.xbs/Sources/XCTest_iOS/XCTest-10112/XCTestFramework/UI Testing/XCUIApplication.m:72
2016-04-03 19:07:40.022 XCTRunner[195:28521] XCTestConfigurationHook: Ignoring skipped tests
2016-04-03 19:07:40.023 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
2016-04-03 19:07:40.025 XCTRunner[195:28521] XCTestConfigurationHook: Using Session ID 75B97C9E-6F14-4F88-A242-64B79C9A0F04
<unknown>:0: error: -[MyTestAppUITests testExample] : failed: caught "NSInternalInconsistencyException", "No target application path specified via test configuration: <XCTestConfiguration: 0x1652af60>
                      testBundleURL:file:///var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns/MyTestAppUITests.xctest/
             testBundleRelativePath:(null)
                  productModuleName:(null)
                        testsToSkip:(null)
                         testsToRun:MyTestAppUITests/testExample
                 reportResultsToIDE:no
                  sessionIdentifier:<__NSConcreteUUID 0x1655e5c0> 75B97C9E-6F14-4F88-A242-64B79C9A0F04
         pathToXcodeReportingSocket:(null)
          disablePerformanceMetrics:no
    treatMissingBaselinesAsFailures:no
                    baselineFileURL:(null)
           baselineFileRelativePath:(null)
              targetApplicationPath:(null)
          targetApplicationBundleID:(null)
                   reportActivities:no
           testsMustRunOnMainThread:no
             initializeForUITesting:no
"
(
    0   CoreFoundation                      0x22c90123 <redacted> + 150
    1   libobjc.A.dylib                     0x22436e17 objc_exception_throw + 38
    2   CoreFoundation                      0x22c8ffe1 <redacted> + 0
    3   Foundation                          0x23463b19 <redacted> + 92
    4   XCTest                              0x002625a1 -[XCUIApplication init] + 216
    5   MyTestAppUITests                    0x01d8a4b7 -[MyTestAppUITests setUp] + 182
    6   XCTest                              0x002433bf __24-[XCTestCase invokeTest]_block_invoke_2 + 198
    7   XCTest                              0x00274661 -[XCTestContext performInScope:] + 308
    8   XCTest                              0x002432f3 -[XCTestCase invokeTest] + 184
    9   XCTest                              0x00243a4f -[XCTestCase performTest:] + 566
    10  XCTest                              0x00241375 -[XCTestSuite performTest:] + 460
    11  XCTest                              0x00241375 -[XCTestSuite performTest:] + 460
    12  XCTest                              0x0024e88d -[XCTestObservationCenter _observeTestExecutionForBlock:] + 716
    13  XCTest                              0x00275c7f _XCTestMain + 1150
    14  CoreFoundation                      0x22c535b1 <redacted> + 12
    15  CoreFoundation                      0x22c5306d <redacted> + 216
    16  CoreFoundation                      0x22c515e1 <redacted> + 784
    17  CoreFoundation                      0x22ba4bf9 CFRunLoopRunSpecific + 520
    18  CoreFoundation                      0x22ba49e5 CFRunLoopRunInMode + 108
    19  GraphicsServices                    0x23df0ac9 GSEventRunModal + 160
    20  UIKit                               0x26e34ba1 UIApplicationMain + 144
    21  XCTRunner                           0x000fa255 XCTRunner + 33365
    22  libdyld.dylib                       0x22853873 <redacted> + 2
)
Test Case '-[MyTestAppUITests testExample]' failed (0.239 seconds).
Test Suite 'MyTestAppUITests' failed at 2016-04-03 19:07:40.256.
     Executed 1 test, with 1 failure (1 unexpected) in 0.239 (0.242) seconds
Test Suite 'Selected tests' failed at 2016-04-03 19:07:40.259.
     Executed 1 test, with 1 failure (1 unexpected) in 0.239 (0.251) seconds

Which to me looks like the MyAppUITests-Runner.app application hasn't included a configuration file which allows it to target the correct application. So I suspect it's something to do with the process I am using to build and upload the application and its UI Tests.

The process I am using to generate the Application ipa file is:

1. In Xcode, click Product > Archive
2. Select the Archive in the Organizer window and click the Export button
3. Select "Save for Ad Hoc Deployment" and click next
4. Select "use local signing assets" for provisioning
5. Select "Export one app for all compatible devices"
6. "Include manifest for over-the-air installation" is UN-checked
7. "Rebuild from bitcode" is checked
8. Export the ipa to a folder on desktop

The process I use to generate the UITests ipa file is (following instructions from http://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest-ui.html):

1. In Xcode, Select "Generic iOS Device" as the build target
2. Click Product > Build For > Testing
3. Open the XCode Derived Data folder for the project, go to Build > Intermediates > Debug-iphoneos. This contains 2 files: MyTestApp.app and MyTestAppUITests-Runner.app
4. Create a new directory called "Payload" on desktop
5. Copy the MyTestAppUITests-Runner.app file into the Payload directory
6. Right click Payload and select "Compress 'Payload'"
7. Rename the generated Payload.zip file to mytestapp-uitests.ipa

I have also tried including both MyTestApp.app and MyTestAppUITests-Runner.app into the mytestapp-uitests.ipa file, but this doesn't work either.

To run the tests in Device Farm:

1. In my testing project, click "Create a new run"
2. Select Android/iOS
3. Upload the MyTestApp.ipa file generated earlier (from Exporting the Archive) and click Next
4. Select "XCTest UI" from the list, and upload the mytestapp-uitests.ipa file

5. Select device pool which consists of iPad Air 2, iPad Air, and iPad 2 all running iOS 9.2.1
6. Leave Device State at default settings
7. Click Start Run

Note that if I use the same application ipa file and run the built-in Fuzz tests, these work with no problems -- I can see the screenshots with my test app with its single label.

I can't see anything obvious which I'm missing. What do I need to do to make this work?

Answer 1

An easier solution to this is to change the location of the Derived data folder - (Under Xcode Preferneces -> Locations.

eg I put it in my User folder ( similar to my documents)

this error went away immediately after battling unsucessfully for hours. ( looks like an Xcode bug)

Answer 2

I work for the AWS Device Farm team. Your analysis quoted below is on the right path.

Which to me looks like the MyAppUITests-Runner.app application hasn't included a configuration file which allows it to target the correct application. So I suspect it's something to do with the process I am using to build and upload the application and its UI Tests.

We are currently fixing an issue where the tests are not picked up if the .xctestconfiguration file is missing from the .ipa.

A workaround for this issue until it is fixed to get you running is to run your tests locally which will generate .xctestconfiguration file under your .app/Plugins folder

Before packaging for upload to device farm just make sure that the .xctestconfiguration file is present under the .app bundle

The Fuzz tests will work because it does not use your test code. It just takes your app and runs a custom fuzz test against it.

Another note about the location of *-Runner.app is you should be able to find it under the Products folder from Xcode but as long as you can get to it things should be fine.

Update This issue has been fixed since the original time of the post and there is no longer a need to have a .xctestconfiguration file in your tests runner.

Answer 3

I have the similar issue. I dig little bit more, it seems the below might be the reason for test to get failed.

"Could not receive XCTest bundle ready message from testmanagerd, error code -7”

It seems the setup and teardown methods are by default run and hence passed. I am not sure about it. But, I will try to get more information from the logs.