eclipse memory analyzer sees small part (363,2MB) of entire heap dump (8GB)

Question

I was trying to investigate java.lang.OutOfMemoryError: GC limit exceeded which occurs at high load of our web app deployed in tomcat. Heap size was set to 8GB (-Xms2048m -Xmx8192m)

At some point in time our application become unresponsive due to GC activity overhead. I could see in logs that Full GC was occurring multiple times in a row. So I took heap dump with following command (jmap -F -dump:format=b,file=/root/dump2.hprof 4963). File containing dump was 9GB in size. After dump was taken (app was frozen for about 45minutes), multiple full GCs occured till OutOfMemoryError was thrown.

Here is a log sample of GC activity

[Full GC [PSYoungGen: 932096K->875513K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6467961K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 12.3954040 secs] [Times: user=47.60 sys=0.43, real=12.39 secs]
[Full GC [PSYoungGen: 932096K->890562K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6483009K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 12.6131900 secs] [Times: user=48.45 sys=0.49, real=12.61 secs]
[Full GC [PSYoungGen: 932096K->895268K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6487715K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 12.9488670 secs] [Times: user=49.61 sys=0.46, real=12.95 secs]

Heap
 PSYoungGen      total 1864128K, used 896698K [0x0000000755560000, 0x0000000800000000, 0x0000000800000000)
  eden space 932096K, 96% used [0x0000000755560000,0x000000078c10e8a8,0x000000078e3a0000)
  from space 932032K, 0% used [0x000000078e3a0000,0x000000078e3a0000,0x00000007c71d0000)
  to   space 932032K, 0% used [0x00000007c71d0000,0x00000007c71d0000,0x0000000800000000)
ParOldGen       total 5592448K, used 5592447K [0x0000000600000000, 0x0000000755560000, 0x0000000755560000)
  object space 5592448K, 99% used [0x0000000600000000,0x000000075555ff30,0x0000000755560000)
PSPermGen       total 262144K, used 112285K [0x00000005e0000000, 0x00000005f0000000, 0x0000000600000000)
  object space 262144K, 42% used [0x00000005e0000000,0x00000005e6da7530,0x00000005f0000000)

heap dump is taken (ca 45minutes freeze)
[Full GC [PSYoungGen: 932096K->903362K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6495810K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 2883.9864390 secs] [Times: user=49.41 sys=0.47, real=2884.17 secs]
[Full GC [PSYoungGen: 932096K->897728K(1864128K)] [ParOldGen: 5592447K->5592444K(5592448K)] 6524543K->6490173K(7456576K) [PSPermGen: 112288K->112288K(262144K)], 13.3092680 secs] [Times: user=50.75 sys=0.40, real=13.31 secs]

To analyze heap dump I opened it in eclipse memory analyzer (MAT). Unfortunately MAT displays that heap size was 363.2MB (in overview tab or heap dump details tab), whereas according to GC logs heap was filled up to 6467961K (6.4G). Unreachable Objects Histogram shows in total 75 511 736 (75 MB). Histogram view also confirmed that total shallow heap was 380 837 136 (363.2MB)

My question is why MAT doesn't display all objects from heap dump if GC cannot reclaim memory?

env details:
Eclipse Memory Analyzer Version 1.2.1
heap dump taken on
java version "1.7.0_13"
Java(TM) SE Runtime Environment (build 1.7.0_13-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Here are screenshots of imported heap dump in MAT:

• overview
• unreachable

Answer 1

By default MAT hides all the Unreachable objects from its views since those objects have already been marked to be garbage collected.

The Unreachable objects can be made accessible in the views following:

1. Close the snapshot
2. Select the heap dump using Window > Heap Dump History right-click, Delete Index Files
3. Select 'Keep unreachable objects' using Window > Preferences > Memory Analyzer
4. Reopen the heap dump, which will reparse the heap dump.
5. Select the Java Basics > GC Roots query.
6. Select the 'Unreachable Objects' row.
7. Run the 'Show Retained Set' query on that row.

This will show a histogram of all the objects which normally would be unreachable and would be garbage collected at the next opportunity. As these objects are now in the snapshot they then can be inspected in more detail.

For detailed info see: Eclipse MAT Reference - Unreachable objects

Answer 2

MAT does not display the unreachable objects by default.

You can enable the option by going to Preferences -> Memory Analyzer -> Keep Unreachable Objects. Load the heap again once the option is enabled.

It will show the complete heap once the option is enabled. Even I was in same situation and was not able to get much information online and my manager showed me the option.Hope it helps.