Menu
I have a large collection of parallel processes that run. It seems that the mere creation of the parallel scala Futures creates a memory leak.
Example code below. Set the VM flag depending on your machine so it doesn't dump right at start in case you have many cores, "-Xmx100m -XX:+HeapDumpOnOutOfMemoryError".
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Await
import scala.concurrent.duration._
object Bug extends App {
println("Memory leak")
while(true){
Future {
val data = new Array[Byte](1000000*1) // 1 MB
println(".")
}
}
}
After a few minutes this code will slow significantly. A heap dump analysis reveals that the array scala.concurrent.forkjoin.ForkJoinTask[] seems to overflow.
The following link seems related but does not really offer any solution to the issue: https://issues.scala-lang.org/browse/SI-7336
364
To avoid memory leaks, memory allocated on heap should always be freed when no longer needed.
DEFINITION A memory leak is the gradual deterioration of system performance that occurs over time as the result of the fragmentation of a computer's RAM due to poorly designed or programmed applications that fail to free up memory segments when they are no longer needed.
Common causes for these memory leaks are: Excessive session objects. Insertion without deletion into Collection objects. Unbounded caches.
Improper terminationIf you terminate your program improperly though, their destructors might not be called, and then memory will be leaked. A common reason for this is using the exit function.
What's happening here is that you're allocating memory much faster then GC`ing it.
In each execution of the while loop you're putting a task to the thread pool to allocate 1MB, you're allocating memory very fast.
After a while GC is what's slowing down.
Solution? Allocate memory slower.
BTW, it will behave in a similar way even if you don't declare the array. When you're running a Future you're submitting a task to the queue of the thread pool so the memory will increase very fast anyway.
To see it yourself you can run jstat (http://www.cubrid.org/blog/dev-platform/how-to-monitor-java-garbage-collection/)
jstat -gc $pid 1000
Here's a sample from my laptop, look how the GCT column (The total accumulated time for GC operations) is increasing
➜ ~ jstat -gc 24504 1000
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT
35328.0 35840.0 22962.9 0.0 275968.0 0.0 395264.0 383080.0 11904.0 11387.3 1664.0 1545.4 30 1.333 3 2.125 3.458
35328.0 35840.0 22962.9 0.0 275968.0 0.0 395264.0 383080.0 11904.0 11387.3 1664.0 1545.4 30 1.333 3 2.125 3.458
35328.0 35840.0 22962.9 0.0 275968.0 0.0 395264.0 383080.0 11904.0 11387.3 1664.0 1545.4 30 1.333 3 2.125 3.458
36352.0 36864.0 8148.0 0.0 275456.0 0.0 562688.0 533891.1 11904.0 11407.6 1664.0 1548.2 36 1.772 4 4.751 6.524
36352.0 36864.0 8148.0 0.0 275456.0 0.0 562688.0 533891.1 11904.0 11407.6 1664.0 1548.2 36 1.772 4 4.751 6.524
36352.0 36864.0 0.0 0.0 275456.0 143402.5 699392.0 486444.1 11904.0 11407.6 1664.0 1548.2 37 1.772 4 6.867 8.640
32768.0 33792.0 22498.3 11986.9 278528.0 278251.8 699392.0 632212.1 11904.0 11437.5 1664.0 1549.8 47 2.548 4 6.867 9.415
29696.0 31232.0 20498.9 0.0 284160.0 0.0 699392.0 678668.1 11904.0 11448.8 1664.0 1551.3 50 2.768 5 6.867 9.635
29696.0 31232.0 20498.9 0.0 284160.0 0.0 699392.0 678668.1 11904.0 11448.8 1664.0 1551.3 50 2.768 5 6.867 9.635
29696.0 31232.0 20498.9 0.0 284160.0 0.0 699392.0 678668.1 11904.0 11448.8 1664.0 1551.3 50 2.768 5 6.867 9.635
29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220
29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220
29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220
29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220
29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220
29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837
29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837
29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837
29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837
29696.0 31232.0 0.0 0.0 284160.0 16821.1 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 7 19.562 22.330
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330
29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880
29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880
29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880
29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880
29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496
29696.0 31232.0 0.0 0.0 284160.0 176242.5 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565
29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
