Object life cycle within a closure

Question

Let's say a closure has been run in JVM and creates an object within the scope of this closure, does this object is still being held in JVM heap space after closure execution while JVM does not terminated?

Once situation is in Spark I use code below to create a HBase connection in an executor:

rdd.foreachPartition(par => {

    # create a connection here.
    val connection = ConnectionFactory.createConnection(conf)

    par.foreach(item => {
        ...
    })

    # close this connection.
    connection.close()
})

Obviously, there connection is created in the executor.

• And after this task has finished, if i never closed this connection, what gonna happen?

• What if spark speculation kills this task while connection is active? will connection is still being held by JVM?

Answer 1

I assume this is not specifically about Spark, but JVM in general.

First of all, a variable created inside a function scope is nothing but a local variable. Hence after the end of the scope, that local variable has no reference and is eligible for GC. So in your particular case, if your connection project doesn't have close() called properly (due to, maybe, some exception happening in between), the connection object is collected by GC but the connection itself isn't closed. We call this situation connection leak.

One of the best practice to deal with this issue is to ensure every connection should get closed, by try...finally, which Java8 have a short hand call try-with-resource. For Scala, one can create an equivalent structure, refer to this post for more details/lesson learned.

For your last question,

What if spark speculation kills this task while connection is active? will connection is still being held by JVM?

JVM would properly not hold the connection any longer. And when this connection isn't closed explicitly by a peer, the other peer (server side) doesn't know if it shouldn't close it accordingly until some idle timeout. Anyway, try finally can cover all the cases to ensure all connections are cleaned up properly.