How can I get Xcode 5 to run iOS unit tests on a Jenkins slave?

Question

We've been using Jenkins with the Xcode plugin to run continuous integration of our iOS applications including automatic execution and reporting of the unit tests. This worked well using Jenkins on a Linux machine and a Mac slave running Xcode 4.6.3.

Now we are looking to update the setup to Xcode 5 to support targetting iOS 7 and have run into the following problems.

First, the unit tests would not run at all because we were using the RunUnitTests script from Xcode 4 that is no longer supported in Xcode 5. I addressed this, as recommended by Xcode, by setting up the workspace with an appropriate scheme with the unit test target.

I then configured the Xcode step of the Jenkins job with custom xcodebuild arguments set to test -destination platform=${DESTINATION_PLATFORM},name=${DESTINATION_NAME},OS=${DESTINATION_OS} to get it to run the unit tests.

If I run the xcodebuild command line that the Xcode plugin for Jenkins runs in Terminal on my own machine, the unit tests run, but when the Jenkins job runs, it either fails or hangs when trying to run the unit tests.

I suspect this is because with Xcode 5 and the test buildaction instead of RunUnitTests script, unit tests now run in the iOS Simulator which requires an interactive session, and the Jenkins slave process is being run over SSH from the Jenkins master (Linux). If I am logged into the slave machine with the account Jenkins uses for SSH, I can see that the iOS Simulator launches when the unit tests are supposed to run, but the tests don't appear to run and the job hangs. If I am not logged into the slave machine, the Jenkins job fails to run the unit tests.

Is there any way to get the iOS unit tests to run on a Jenkins slave over SSH, and if not, any suggestions on how to keep automating the execution of the unit tests when the project must be built with Xcode 5?

Answer 1

Based on coffeebreaks' answer, I've come up with a full solution.

First of all, the Mac slave cannot be launched with SSH and must be launched manually using an interactive session and then left logged in at all times. In my situation, the slave is actually headless, so this is a further complication.

Here are the steps I used to get this all operational.

1. Create a new slave node on the Jenkins master configured with a unique label (I chose "xcode-unittests") and launch method set to "Launch slave agents via Java Web Start).

2. Login via screen sharing (VNC) to the Mac slave and start the slave agent. In my case, I could not get the slave to launch from the browser, probably because my browser did not have the necessary Java plug-in for running applets. I therefore used the command line javaws http://{jenkins-host}/computer/{slave-name}/slave-agent.jnlp. To make this a bit more robust, I configured this command to automatically run every time I login to the machine interactively, under System Preferences, Users & Groups, Login Items.

3. Quit Screen Sharing without logging out of the Mac slave. This keeps the interactive session running the slave agent alive, even though nobody is actually using the machine.

In order for the unit tests to run without prompting a user, I also had to run sudo DevToolsSecurity -enable on the Mac slave. This allows Xcode to interact with the iOS Simulator without interactively asking for permission to do so each time.

If the slave machine is ever rebooted, someone must login to the Mac slave to get the slave agent to run again. For that reason, I also left my SSH-based slave active as well. I split my Jenkins jobs into separate jobs for building the apps and running the unit tests. The jobs for building the apps are configured to run on the SSH-based slave, and the jobs for running the unit tests are configured to run on the interactive slave node described above. That way, if the interactive slave goes down, only the unit tests are affected, not the product builds.

Answer 2

Try starting your jenkins slave from a standard terminal on your Mac slave, not from the master using SSH.

Answer 3

See GBegen and coffeebreaks' answers. When the Jenkins is secured, use the alternative command line to start it from a screen share/VNC terminal session is

java -jar slave.jar -jnlpUrl http://jenkins-master:port/computer/jenkins-slave/slave-agent.jnlp -secret XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX