How to avoid fast memory increase during scavenge gc?

Question

I have an application built on restify. I have no memory leaks, however I have big memory growth during scavenge gc, then comes heavy weight mark-sweep gc and cleans the memory.

It affects performance of my application.

[2268]   266859 ms: Scavenge 61.5 (119.5) -> 46.0 (119.5) MB, 2.2 ms [allocation failure].
[2268]   267084 ms: Scavenge 63.7 (119.5) -> 48.3 (119.5) MB, 6.2 ms [allocation failure].
[2268]   267289 ms: Scavenge 66.0 (119.5) -> 50.6 (119.5) MB, 2.6 ms [allocation failure].
[2268]   267504 ms: Scavenge 68.3 (119.5) -> 52.8 (119.5) MB, 2.4 ms [allocation failure].
[2268]   267700 ms: Scavenge 70.5 (119.5) -> 55.1 (119.5) MB, 2.7 ms [allocation failure].
....

[2268]   275913 ms: Scavenge 154.2 (183.5) -> 138.8 (183.5) MB, 2.4 ms [allocation failure].
[2268]   276161 ms: Scavenge 157.5 (185.5) -> 142.1 (185.5) MB, 2.7 ms (+ 2.4 ms in 18 steps since last GC) [allocation failure].
[2268]   276427 ms: Scavenge 159.8 (187.5) -> 144.3 (187.5) MB, 2.5 ms (+ 36.3 ms in 236 steps since last GC) [allocation failure].
[2268]   276494 ms: Mark-sweep 147.7 (188.5) -> 43.7 (121.5) MB, 20.1 ms (+ 45.1 ms in 298 steps since start of marking, biggest step 0.5 ms) [GC interrupt] [GC in old space requested].

This type of behavior happens when I try to access non-existing url

ab -c 100 -n 10000000 -k http://localhost:1337/invalid/url

I can't really use node-inspector to track what causes such intense memory growth because it will request full gc before taking heap snapshot.

What are my options to track what causes such a rapid memory growth?

How to find out which objects survive scavenges but don't survive mark-sweep gc?

Thanks,

UPDATE 1 So there is no way to see middle aged scavenged content. Here is the hint, if you see fast memory increase during scavenge but then it suddenly drops with mark&sweep then it's mean that your code creates data in large space. Long stack traces for example. Restify generates gigantic stack traces which should be disabled in production.

Answer 1

You might try running your Node script with the option -–expose-gc:

node --expose-gc script.js

This allows to trigger the garbage collection manually from within JS:

global.gc();

When garbage collection is enforced manually, you can apply a multiple snapshot technique:

• take one Snap before, one Snap after GC
• then apply optimization
• then one Snap before, one Snap after GC

The snapshots allow to track what causes the memory growth. The goal is to have a better result of the second "Snap after GC", when compared to the first "Snap after GC".