Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

How to prevent Scala Futures from creating a memory leak

Tags:

memory-leaks

scala

I have a large collection of parallel processes that run. It seems that the mere creation of the parallel scala Futures creates a memory leak.

Example code below. Set the VM flag depending on your machine so it doesn't dump right at start in case you have many cores, "-Xmx100m -XX:+HeapDumpOnOutOfMemoryError".

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Await
import scala.concurrent.duration._

object Bug extends App {
    println("Memory leak")
    while(true){
        Future {
            val data = new Array[Byte](1000000*1) // 1 MB
            println(".")
        }
    }
}

After a few minutes this code will slow significantly. A heap dump analysis reveals that the array scala.concurrent.forkjoin.ForkJoinTask[] seems to overflow.

The following link seems related but does not really offer any solution to the issue: https://issues.scala-lang.org/browse/SI-7336

like image 364
calloc_org Avatar asked Jan 01 '16 05:01

calloc_org

People also ask

How would you prevent a memory leak?

To avoid memory leaks, memory allocated on heap should always be freed when no longer needed.

What is the main cause of memory leaks?

DEFINITION A memory leak is the gradual deterioration of system performance that occurs over time as the result of the fragmentation of a computer's RAM due to poorly designed or programmed applications that fail to free up memory segments when they are no longer needed.

Which action can cause memory leak?

Common causes for these memory leaks are: Excessive session objects. Insertion without deletion into Collection objects. Unbounded caches.

Does exit cause memory leaks?

Improper terminationIf you terminate your program improperly though, their destructors might not be called, and then memory will be leaked. A common reason for this is using the exit function.

1 Answers

What's happening here is that you're allocating memory much faster then GC`ing it.

In each execution of the while loop you're putting a task to the thread pool to allocate 1MB, you're allocating memory very fast.

After a while GC is what's slowing down.

Solution? Allocate memory slower.

BTW, it will behave in a similar way even if you don't declare the array. When you're running a Future you're submitting a task to the queue of the thread pool so the memory will increase very fast anyway.

To see it yourself you can run jstat (http://www.cubrid.org/blog/dev-platform/how-to-monitor-java-garbage-collection/)

jstat -gc $pid 1000

Here's a sample from my laptop, look how the GCT column (The total accumulated time for GC operations) is increasing 

➜  ~  jstat -gc 24504 1000
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
35328.0 35840.0 22962.9  0.0   275968.0   0.0     395264.0   383080.0  11904.0 11387.3 1664.0 1545.4     30    1.333   3      2.125    3.458
35328.0 35840.0 22962.9  0.0   275968.0   0.0     395264.0   383080.0  11904.0 11387.3 1664.0 1545.4     30    1.333   3      2.125    3.458
35328.0 35840.0 22962.9  0.0   275968.0   0.0     395264.0   383080.0  11904.0 11387.3 1664.0 1545.4     30    1.333   3      2.125    3.458
36352.0 36864.0 8148.0  0.0   275456.0   0.0     562688.0   533891.1  11904.0 11407.6 1664.0 1548.2     36    1.772   4      4.751    6.524
36352.0 36864.0 8148.0  0.0   275456.0   0.0     562688.0   533891.1  11904.0 11407.6 1664.0 1548.2     36    1.772   4      4.751    6.524
36352.0 36864.0  0.0    0.0   275456.0 143402.5  699392.0   486444.1  11904.0 11407.6 1664.0 1548.2     37    1.772   4      6.867    8.640
32768.0 33792.0 22498.3 11986.9 278528.0 278251.8  699392.0   632212.1  11904.0 11437.5 1664.0 1549.8     47    2.548   4      6.867    9.415
29696.0 31232.0 20498.9  0.0   284160.0   0.0     699392.0   678668.1  11904.0 11448.8 1664.0 1551.3     50    2.768   5      6.867    9.635
29696.0 31232.0 20498.9  0.0   284160.0   0.0     699392.0   678668.1  11904.0 11448.8 1664.0 1551.3     50    2.768   5      6.867    9.635
29696.0 31232.0 20498.9  0.0   284160.0   0.0     699392.0   678668.1  11904.0 11448.8 1664.0 1551.3     50    2.768   5      6.867    9.635
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 16821.1   699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   7     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 176242.5  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
like image 126
Maxim Avatar answered Nov 07 '22 12:11

Maxim
Sign in to Comment

Related questions

Unresolved dependency with specs2 scalaz-stream 0.5a
Parboiled2 causes "missing or invalid dependency detected while loading class file 'Prepender.class'"
Using `title` with ScalaTags
json4s jackson - How to ignore field using annotations
AWS S3: Uploading large file fails with ResetException: Failed to reset the request input stream
Convert Matrix to RowMatrix in Apache Spark using Scala
How to find the name of the enclosing source file in Scala 2.11
Difference between Scala REPL and Clojure REPL - compile speed
Akka scheduling patterns
How do I compose a list of `Futures`?
Caused by: java.sql.SQLException: JDBC4 Connection.isValid() method not supported
Using v. Not Using the `self` Type
Parsing a Json String in Scala using Play framework
Akka cluster detecting Quarantined state
Understanding closures and parallelism in Spark
Idiomatic way to use Spark DStream as Source for an Akka stream
Scala compiler optimization for immutability
accumulator of Spark is confusing me.
Spark: group concat equivalent in scala rdd
In Apache spark, what is the difference between using mapPartitions and combine use of broadcast variable and map

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!