Why does the first run of "XCTestCase -measureBlock:" takes so much time?

Question

I'm using Xcode 7. iPhone 5s simulator (iOS 9.0).

I'm trying to use XCTestCase -measureBlock: to measure time spent on some of my functions. This runs the code inside the block 10 times and report the results how long each run takes, the average, STDEV, etc.

It turns out that the time of first run is always very high, like 400-500% higher. Same happens for a very simple method, or even nothing in the measure block. I don't have anything in my - (void)setUp or - (void)tearDown.

Result of NSString -stringWithFormat:. See the spike at the first bar of the graph. values: [0.000031, 0.000005, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003, 0.000003]

Result of a blank block. Similar results. values: [0.000007, 0.000001, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000001]

Is this a bug or something? What is going on?

Answer 1

It's probably an artifact of some caches warming up.

The Objective-C runtime maintains a selector cache for each class, for example. This means that the first call to each method is somewhat slower than the following calls.

You could just put the contents of the measured block right before the call to measureBlock: to test if this is true. So the warm up run would just not be measured, then.

Answer 2

The measured block is executed ten times and the test output shows the average execution time as well as individual run times and standard deviation (http://nshipster.com/xctestcase/) So first time it can populate cache and prepare data if your code does it, on rest next steps it might work with cached data and execution time can be shorter as well.