Kotlin case of non-intuitive type inference

Question

I found some non-intuitive behavior of type inference. As a result, the semantically equivalent code works differently, depending on what information the compiler infers about function return type. It is more or less clear what is going on when you reproduce this case in a minimum unit test. But I afraid that when writing framework code, such behavior could be dangerous.

The code below illustrates the problem, and my questions are:

1. Why the puzzler1 call from notok1 unconditionally throws NPE? As far as I understand from the bytecode, ACONST_NULL ATHROW throws NPE right after puzzler1 call, ignoring the returned value.

2. Is it normal that upper bound (<T : TestData>) is ignored when compiler infers the type?

3. Is it a bug that NPE becomes ClassCastException if you add suspend modifier to the function? Of course, I understand that runBlocking+suspend call gives us the different bytecode, but shouldn't the "coroutinized" code be as equivalent as possible to conventional code?

4. Is there a way to rewrite puzzler1 code somehow, eliminating the unclearness?

@Suppress("UnnecessaryVariable", "MemberVisibilityCanBePrivate", "UNCHECKED_CAST", "RedundantSuspendModifier")
class PuzzlerTest {

    open class TestData(val value: String)

    lateinit var whiteboxResult: TestData

    fun <T : TestData> puzzler1(
        resultWrapper: (String) -> T
    ): T {
        val result = try {
            resultWrapper("hello")
        } catch (t: Throwable) {
            TestData(t.message!!) as T
        }
        whiteboxResult = result
        return result // will always return TestData type
    }

    // When the type of `puzzler1` is inferred to TestData, the code works as expected:
    @Test
    fun ok() {
        val a = puzzler1 { TestData("$it world") }
        // the same result inside `puzzler1` and outside of it:
        assertEquals("hello world", whiteboxResult.value)
        assertEquals("hello world", a.value)
    }

    // But when the type of `puzzler1` is not inferred to TestData, the result is rather unexpected.
    // And compiler ignores the upper bound <T : TestData>:
    @Test
    fun notok1() {
        val a = try {
            puzzler1 { throw RuntimeException("goodbye") }
        } catch (t: Throwable) {
            t
        }
        assertEquals("goodbye", whiteboxResult.value)
        assertTrue(a is NullPointerException) // this is strange
    }

    // The same code as above, but with enough information for the compiler to infer the type:
    @Test
    fun notok2() {
        val a = puzzler1 {
            @Suppress("ConstantConditionIf")
            if (true)
                throw RuntimeException("goodbye")
            else {
                // the type is inferred from here
                TestData("unreachable")

                // The same result if we write:
                // puzzler1<TestData> { throw RuntimeException("goodbye") }
            }
        }

        assertEquals("goodbye", whiteboxResult.value)
        assertEquals("goodbye", (a as? TestData)?.value) // this is stranger
    }

    // Now create the `puzzler2` which only difference from `puzzler1` is `suspend` modifier:

    suspend fun <T : TestData> puzzler2(
        resultWrapper: (String) -> T
    ): T {
        val result = try {
            resultWrapper("hello")
        } catch (t: Throwable) {
            TestData(t.message!!) as T
        }
        whiteboxResult = result
        return result
    }

    // Do exactly the same test as `notok1` and NullPointerException magically becomes ClassCastException:
    @Test
    fun notok3() = runBlocking {
        val a = try {
            puzzler2 { throw RuntimeException("goodbye") }
        } catch (t: Throwable) {
            t
        }
        assertEquals("goodbye", whiteboxResult.value)
        assertTrue(a is ClassCastException) // change to coroutines and NullPointerException becomes ClassCastException
    }

    // The "fix" is the same as `notok2` by providing the compiler with info to infer `puzzler2` return type:
    @Test
    fun notok4() = runBlocking {
        val a = try {
            puzzler2<TestData> { throw RuntimeException("goodbye") }

            // The same result if we write:
            // puzzler2 {
            //     @Suppress("ConstantConditionIf")
            //     if (true)
            //         throw RuntimeException("goodbye")
            //     else
            //         TestData("unreachable")
            // }
        } catch (t: Throwable) {
            t
        }
        assertEquals("goodbye", whiteboxResult.value)
        assertEquals("goodbye", (a as? TestData)?.value)
    }
}

Answer 1

What is the type of throw RuntimeException("goodbye")? Well, since it never returns a value, you can use it anywhere you like, no matter what type of object is expected, and it will always typecheck. We say that it has type Nothing. This type has no values, and it is a subtype of every type. Therefore, in notok1, you have a call to puzzler1<Nothing>. The cast from the constructed TestData to T = Nothing inside puzzler1<Nothing> is unsound but unchecked, and puzzler1 ends up returning when its type signature says it shouldn't be able to. notok1 notices that puzzler1 has returned when it said it would not be able to, and immediately throws an exception itself. It's not very descriptive, but I believe the reason it throws an NPE is because something has gone "terribly wrong" if a function that can't return has returned, so the language decides the program should die as fast as possible.

For notok2, you actually do get T = TestData: one branch of the if returns Nothing, the other TestData, and the LUB of those is TestData (since Nothing is a subtype of TestData). notok2 has no reason to believe that puzzler1<TestData> cannot return, so it doesn't set up the trap to die as soon as puzzler1 returns.

notok3 has essentially the same problem as notok1. The return type, Nothing, implies that the only thing the puzzler2<Nothing> will do is throw an exception. The coroutine handling code in notok3 thus expects the coroutine to hold a Throwable and contains code to rethrow it, but does not contain code to handle an actual return value. When puzzler2 actually does return, notok3 tries to cast that TestData into a Throwable and fails. notok4 works for the same reason notok2 does.

The solution to this mess is simply not using an unsound cast. Sometimes puzzler1<T>/puzzler2<T> will be able to return a T, if the passed function in fact returns a T. But, if that function throws, they can only return a TestData, and a TestData is not a T (a T is a TestData, not the other way around). The correct signature for puzzler1 (and similarly for puzzler2) is

fun <T : TestData> puzzler1(resultWrapper: (String) -> T): TestData

Since functions are covariant in the return type, you can just get rid of the type parameter

fun puzzler1(resultWrapper: (String) -> TestData): TestData