Metaspace memory concumption issues of Java 8 applications deployed on Wildfly 8.2.1

Question

Issue description

I noticed that each deployment of one of our Java8 application on Wildfly 8.2.1 uses approximately 30-40 MBs from Metaspace memory pool. That is fine, but the thing is that once I am redeploying the same app the Metaspace memory usage is being increased by the same 30-40 MBs whereas the old already allocated memory is not being released.

I would not even notice it but the thing is that we have ~20 applications and from time to time I need to redeploy up to 10 of them at the same time. That in turn results in a scary picture.

Basically what is shown are the 2 redeployments of ~10 applications.

I'm not sure why GC cannot release the memory allocated to the old classes. This server has in total 16GB physical memory so I can redeploy all apps up to 20-40 times and that is it. App server will reach the limit and will stop respond to any commands.

So I would be very thankful if someone could help me to understand what the actual issue could be:

1. Is it a Java8 issue? (jdk 1.8.0_40-b26)
2. Is it a Wildfly 8.2.1 issue?
3. Or is the answer as simple as possible that the cause is my code base? If so, then could you please guide me what could be the actual reason?

Some more details related to my code base

1) Together with Wildfly I use 2 standalone HornetQ servers, each application uses ~5 channels with at least 5 concurrent consumers on each. Which in turn results in at least 25 threads for each app and at least 25*20 = 500 threads in total.

2) For all low-level JMS operations I use Spring JMS.

Answer 1

To empirically determine where and if you have a leak (you may not have a leak - it could just be legitimate classes getting loaded during deployments for legitimate reasons) you might try taking a heap dump at the right time (i.e. before or after the increase in the metaspace occurs). Then, take another heap dump. Diff the two heap dumps using a tool like MAT or Yourkit. The diff between the two will tell you which classes are being loaded.

Leaks in the metaspace are pretty rare because you only have so many classes to load but it could be something pretty exotic at play as well.

Let me know what you find and happy hunting! These are fun. :-)

Answer 2

WildFly 10.0.0.Final "java.lang.OutOfMemoryError: Metaspace" occurs and will be fixed. Refer to follow Agile Board of Wildfly

https://issues.jboss.org/browse/WFLY-6173