SSL Connections Issues when running Unit Tests from the command line

Question

Goal

Our goal is to execute our Unit Tests within a Continuous Integration environment (Jenkins)

(I believe it is essential for every question to state what exactly one is trying to achieve. Maybe the problem can actually be solved a very different way)

Update: More on "Why do you want to do this?"

Firstly, we are talking to specific, self-developed hardware. And I would like the Continuous Integration Test to tell me if anyone changed the behavior of the box without telling all the developers (Yes, yes, I know, such things never happen …)

Secondly, some (not all) of our connections use certificates which are not valid by default, so we have code to check the validity of the certificate (SecTrustEvaluate etc.). Ideally of course, our tests would test that code, too. But that seems too much to ask.

Thirdly: Well, why should I mock anything if I can have the real deal? The IDE has no issue with that, why should the command line.

And if I wanted predictability, I'd return YES in all my tests ;) (and I have seen people do that). No, I want to know if the code actually and really works with our device. Is that some sort of perverted desire?

Problem resolved so far

Running tests from the command line sounds pretty straighforward, but

xcodebuild -sdk iphonesimulator ONLY_ACTIVE_ARCH=NO clean test

results in an ugly error:

unsupported build action 'test'

So I searched the web and found this article on Running OCUnit Tests from Command Line.

I followed all the steps, and I can run my tests from the command line like this:

xcodebuild -scheme CITests -sdk iphonesimulator TEST_AFTER_BUILD=YES ONLY_ACTIVE_ARCH=NO clean build

Remaining Problem

However, now any NSURLConnection to an SSL Server will fail, because "The certificate for this server is invalid.". I have heared about keychain issues when running tests from the command line, but can this be true? Any SSL connection is refused?

To Reproduce

Get the sample at https://github.com/below/SSLTestDemo. Open it up, and run the sample test using Xcode's own Test ⌘U command. The test should succeed.

Now run the test on the command line:

xcodebuild -scheme CITests -sdk iphonesimulator TEST_AFTER_BUILD=YES ONLY_ACTIVE_ARCH=NO clean build

The test fails due to the "The certificate for this server is invalid." error.

Any Pointers?

• Am I doing it wrong?
• Is this a genuine bug?
• If so, are there workaroungs?

Any input is appreciated!

Answer 1

Well, I ended up simply not checking certificates when we are testing under Jenkins.

As it is dangerous to have code which prevents such checking in your app — after all, it just might not get removed before you ship — you have to set both an environment variable and a compiler flag to activate it …

Post Scriptum:

Apple considers this a bug, rdar://problem/10406441