How to test generic performance with whole module optimization

Question

In WWDC 2015 Session 409 near the 18 minute mark. The discussion at hand leads me to believe that generics can be optimized through Generic Specialization by enabling whole module optimization mode. Unfortunately my tests, which I'm not confident in, revealed nothing useful.

I ran some very simple tests between the following two methods to see if the performance was similar:

func genericMax<T : Comparable>(x:T, y:T) -> T {
    return y > x ? y : x
}

func intMax(x:Int, y:Int) -> Int {
    return y > x ? y : x
}

Simple XCTest:

func testPerformanceExample() {

    self.measureBlock {

        let x: Int = Int(arc4random_uniform(9999))
        let y: Int = Int(arc4random_uniform(9999))

        for _ in 0...1000000 {
            // let _ = genericMax(x, y: y)
            let _ = intMax(x, y: y)
        }
    }
}

What happened

Without optimization the following tests were reasonably different:

• genericMax: 0.018 sec
• intMax: 0.005 sec

However with Whole Module Optimization the following tests weren't similar:

• genericMax: 0.014 sec
• intMax: 0.004 sec

What I Expected

With whole module optimization enabled I expected similar times between the two methods calls. This leads me to believe that my test is flawed.

Question

Assuming my tests are flawed / poor. How could I better measure how Whole Module Optimization mode optimizes generics through Generic Specialization?

Answer 1

Your tests are flawed because they measure test performance, not app performance. Tests live in a separate executable file, and they do not benefit from whole-module optimizations themselves. Because of this, your test always uses the generic, non-specialized implementation even in places where your program doesn't.

If you want to see that whole-module optimizations are enabled in your executable, you need to test a function from your executable. (You also need to make sure that your tests either use the Release build, or that you have WMO enabled in the debug build.)

Adding this function to the executable:

func genericIntMax(x: Int, y: Int) -> Int {
    return genericMax(x, y: y)
}

and using it from the tests in place of genericMax in the tests, I get identical performance. (Note that this is not really whole module optimization since the two functions live in the same file; but it shows the difference between app code and test code when it comes to optimizations.)