Jenkins on OS X: xcodebuild gives Code Sign error

Question

Summary:

Setting up Jenkins on OS X has been made significantly easier with the most recent installer (as of 1.449 - March 9, 2012), however managing the process of code signing is still very difficult with no straightforward answer.

Motivation:

Run a headless CI server that follows common best practices for running services on OS X (Some of which is explained here in plain language).

Background:

• October 12, 2009 - How to automate your iPhone app builds with Hudson
• June 15, 2011 - Jenkins on Mac OS X; git w/ ssh public key
• June 23, 2011 - Continuous Deployment of iOS Apps with Jenkins and TestFlight
• July 26, 2011 - Missing certificates and keys in the keychain while using Jenkins/Hudson as Continuous Integration for iOS and Mac development
• August 30, 2011 - Xcode Provisioning File not found with Jenkins
• September 20, 2011 - How to set up Jenkins CI on a Mac
• September 14, 2011 - Getting Jenkins Running on a Mac
• November 12, 2011 - Howto: Install Jenkins on OS X and make it build Mac stuff
• January 23, 2012 - Upcoming Jenkins OSX installer changes
• March 7, 2012 - Thanks for using OSX Installer

Process:

Install Jenkins CI via OS X installer package. For the "Installation Type" step, click the Customize button, and choose "Start at boot as 'jenkins.'"

Discussion:

The naive expectation at this point was that a free-style project with the build script xcodebuild -target MyTarget -sdk iphoneos should work. As indicated by the title of this post, it does not and fails with:

Code Sign error: The identity 'iPhone Developer' doesn't match any valid certificate/private key pair in the default keychain

It is obvious enough what needs to happen - you need to add a valid code signing certificate and a private key into the default keychain. In researching how to accomplish this, I have not found a solution that doesn't open up the system to some level of vulnerability.

Problem 1: No default keychain for jenkins daemon

sudo -u jenkins security default-keychain ...yields "A default keychain could not be found"

As pointed out below by Ivo Dancet, the UserShell is set to /usr/bin/false for the jenkins daemon by default (I think this is a feature, not a bug); follow his answer to change the UserShell to bash. You can then use sudo su jenkins to get logged in as the jenkins user and get a bash prompt.

1. sudo su jenkins
2. cd ~/Library
3. mkdir Keychains
4. cd Keychains
5. security create-keychain <keychain-name>.keychain
6. security default-keychain -s <keychain-name>.keychain

Okay, great. We've got a default keychain now; let's move on right? But, first why did we even bother making a default keychain?

Almost all answers, suggestions, or conversation I read throughout researching suggest that one should just chuck their code signing certs and keys into the system keychain. If you run security list-keychains as a free-style project in Jenkins, you see that the only keychain available is the system keychain; I think that's where most people came up with the idea to put their certificate and key in there. But, this just seems like a very bad idea - especially given that you'll need to create a plain text script with the password to open the keychain.

Problem 2: Adding code signing certs and private key

This is where I really start to get squeamish. I have a gut feeling that I should create a new public / private key unique for use with Jenkins. My thought process is if the jenkins daemon is compromised, then I can easily revoke the certificate in Apple's Provisioning Portal and generate another public / private key. If I use the same key and certificate for my user account and Jenkins, then it means more hassle (damage?) if the jenkins service is attacked.

Pointing to Simon Urbanek's answer you'll be unlocking the keychain from a script with a plain text password. It seems irresponsible to keep anything but "disposable" certificates and keys in the jenkins daemon's keychain.

I am very interested in any discussion to the contrary. Am I being overly cautious?

To make a new CSR as the jenkins daemon in Terminal I did the following...

1. sudo su jenkins
2. certtool r CertificateSigningRequest.certSigningRequest You'll be prompted for the following (most of these I made educated guesses at the correct answer; do you have better insight? Please share.)...
   • Enter key and certificate label:
   • Select algorithm: r (for RSA)
   • Enter key size in bits: 2048
   • Select signature algorithm: 5 (for MD5)
   • Enter challenge string:
   • Then a bunch of questions for RDN
3. Submit the generated CSR file (CertificateSigningRequest.certSigningRequest) to Apple's Provisioning Portal under a new Apple ID
4. Approve the request and download the .cer file
5. security unlock-keychain
6. security add-certificate ios_development.cer

This takes us one step closer...

Problem 3: Provisioning profile and Keychain unlocking

I made a special provisioning profile in the Provisioning Portal just for use with CI in hopes that if something bad happens I've made the impact a little smaller. Best practice or overly cautious?

1. sudo su jenkins
2. mkdir ~/Library/MobileDevice
3. mkdir ~/Library/MobileDevice/Provisioning\ Profiles
4. Move the provisioning profile that you setup in the Provisioning Portal into this new folder. We're now two short steps away from being able to run xcodebuild from the the command line as jenkins, and so that means we're also close to being able to get the Jenkins CI running builds.
5. security unlock-keychain -p <keychain password>
6. xcodebuild -target MyTarget -sdk iphoneos

Now we get a successful build from a command line when logged in as the jenkins daemon, so if we create a free-style project and add those final two steps (#5 and #6 above) we will be able to automate the building of our iOS project!

It might not be necessary, but I felt better setting jenkins UserShell back to /usr/bin/false after I'd successfully gotten all this setup. Am I being paranoid?

Problem 4: Default keychain still not available!

(EDIT: I posted the edits to my question, rebooted to make sure my solution was 100%, and of course, I'd left out a step)

Even after all the steps above, you'll need to modify the Launch Daemon plist at /Library/LaunchDaemons/org.jenkins-ci.plist as stated in this answer. Please note this is also an openrdar bug.

It should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>EnvironmentVariables</key>
        <dict>
                <key>JENKINS_HOME</key>
                <string>/Users/Shared/Jenkins/Home</string>
        </dict>
        <key>GroupName</key>
        <string>daemon</string>
        <key>KeepAlive</key>
        <true/>
        <key>Label</key>
        <string>org.jenkins-ci</string>
        <key>ProgramArguments</key>
        <array>
                <string>/bin/bash</string>
                <string>/Library/Application Support/Jenkins/jenkins-runner.sh</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>UserName</key>
        <string>jenkins</string>
        <!-- **NEW STUFF** -->
        <key>SessionCreate</key>
        <true />
</dict>
</plist>

With this setup, I would also recommend the Xcode plugin for Jenkins, which makes setting up the xcodebuild script a little bit easier. At this point, I'd also recommend reading the man pages for xcodebuild - hell you made it this far in Terminal, right?

This setup is not perfect, and any advice or insight is greatly appreciated.

I have had a hard time selecting a "correct" answer since what I've come to use to solve my problem was a collection of just about everyone's input. I've tried to give everyone at least an up vote, but award the answer to Simon because he mostly answered the original question. Furthermore, Sami Tikka deserves a lot of credit for his efforts getting Jenkins to work through AppleScript as a plain ol' OS X app. If you're only interested in getting Jenkins up and going quickly within your user session (i.e. not as a headless server) his solution is much more Mac-like.

I hope that my efforts spark further discussion, and help the next poor soul who comes along thinking they can get Jenkins CI setup for their iOS projects in a weekend because of all the wonderful things they've heard about it.

---

Update: August 9, 2013

With so many upvotes and favorites, I thought I would come back to this 18 months later with some brief lessons learned.

Lesson 1: Don't expose Jenkins to the public internet

At the 2012 WWDC I took this question to the Xcode and OS X Server engineers. I received a cacophony of "don't do that!" from anyone I asked. They all agreed that an automated build process was great, but that the server should only be accessible on the local network. The OS X Server engineers suggested allowing remote access via VPN.

Lesson 2: There are new install options now

I recently gave a CocoaHeads talk about my Jenkins experience, and much to my surprise I found some new install methods - Homebrew and even a Bitnami Mac App Store version. These are definitely worth checking out. Jonathan Wright has a gist detailing getting Homebrew Jenkins working.

Lesson 3: No, seriously, don't expose your build box to the internet

It's pretty clear from the original post that I'm neither a system administrator nor security expert. Common sense about private-y stuff (keychains, credentials, certificates, etc) left me feeling pretty uneasy about putting my Jenkins box on the internet. Nick Arnott at Neglected Potential was able to confirm my heebie-jeebies pretty easily in this article.

TL;DR

My recommendation to others looking to automate their build process has changed over the past year and a half. Make sure your Jenkins machine is behind your firewall. Install and set Jenkins up as a dedicated Jenkins user either using the installer, Bitnami Mac App Store version, Sami Tikka's AppleScript, etc; this resolves most of the headache I detail above. If you need remote access, setting up VPN services in OS X Server takes ten minutes tops. I've been using this setup for over a year and am very happy with it. Good luck!

Answer 1

Keychains need to be unlocked before they can be used. You can use security unlock-keychain to unlock. You can do that interactively (safer) or by specifying the password on the command line (unsafe), e.g.:

security unlock-keychain -p mySecretPassword...

Obviously, putting this into a script compromises the security of that keychain, so often people setup an individual keychain with only the signing credentials to minimize such damage.

Typically in Terminal the keychain is already unlocked by your session, since the default keychain is unlocked on login, so you don't need to do that. However, any process not run in your session won't have unlocked keychain even if it has you as the user (most commonly this affects ssh, but also any other process).

Answer 2

Suppose you also want to do ad hoc distribution through Jenkins, this necessitates that Jenkins has access to a Distribution certificate, and the team admin identity, in addition to the provisioning profiles.

Using an exported identity in a .cer file, you can programmatically import it like so, the -A switch is to allow all programs access to this entry. Alternatively, you could use several -T /path/to/program switches to allow codesign and xcodebuild access.:

$ security import devcertificate.cer -k jenkins.keychain -A

Of course, we should also have the Apple WWDCRA certificate, imported in pretty much the same way:

$ security import AppleWWDRCA.cer -k jenkins.keychain -A

However, we also need the private key for the devcertificate.cer. To do this, you need to export the corresponding private key as a .p12 key and set a password. Put it somewhere you can access it from your Jenkins shell, unlock the keychain, and import it:

$ security unlock-keychain -p YourKeychainPass jenkins.keychain
$ security import devprivatekey.p12 -k login.keychain -P ThePasswordYouSetWhenExporting -A

Importing the distribution certificate works the same way. I don't know why you need to unlock the keychain for importing a .p12 and not for a .cer, but well.

You will also need access to the provisioning profiles, I will edit those instructions into this post shortly.

Answer 3

I've had the same issue and have been searching around for some time for an answer. Here's one thing that I've learned.

I am running jenkins as the jenkins user, user created by the installer, and as everyone else has mentioned he doesn't have access to the same keychain that your normal user does. Instead of trying to login as the jenkins user, I created a second build project that simply has one build step that is "Execute Shell" in which I run commands I want to test as the jenkins user.

Once I had that set up, I could run the command

security list-keychains

And this revealed to me that the only thing that jenkins could see was the system keychain.

+ security list-keychains
    "/Library/Keychains/System.keychain"
    "/Library/Keychains/System.keychain"

With that knowledge, I then opened the Keychain Access app and copied my "iPhone Developer: xxxx" certificate into the System keychain (Right-click, copy from the "login" keychain).

This got me passed the certificate/private key pair code sign error but opened up another one with the provisioning profile (seems like a similar, but different, issue).