Memory leak in Scala and processes

Question

I have a system in Scala, with a lot of simultaneous threads and system calls. This system has some problem, because memory usage is increasing over time.

The image bellow shows the memory usage for one day. When it gets to the limit, the process shuts down and I put a watch-dog to recover it again.

I periodically run the command

jcmd <pid> GC.run

And this makes the memory to increase slowly, but the leak still happens.

I analysed with jvisualvm, comparing to distinct moments in time, with 40 minutes delta. The image bellow shows the comparison between these two moments in time. Notice that there is an increase for instances of some classes like ConcurrentHashMap$HashEntry, SNode, WeakReference, char[] and String and many classes in the package scala.collection.concurrent.

What can be causing the memory leak?

Edit 1: Investigating JVisualVM, I noticed object of CNode and INode classes that are in TriedMap, that is instanced inside sbt.TrapExit$App class. Here is the object hierarchy figure:

Answer 1

First capture a heap dump when your application crashes due to an out of memory issue. Add the following flags when starting the jvm

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dump

Next you need to analyze the heap dump to figure out the source of the memory leak. I recommend using Eclipse MAT. The Leak Suspects report should give you a sense of what objects are actually causing the leak.