Memory leaks in Node.js - How to analyze allocation tree/roots?

Question

Finding memory leaks is a very difficult task, especially when it comes to modern JS code that makes use of many third party libraries. For example, I am currently facing down a memory leak in rollup, involving babel and a custom babel plugin. I am exploring several common strategies to hunting them down:

1. Understand your runtime, its memory de-allocation scheme, and follow best practices regarding that scheme.
   • This article claims that all modern JS runtime implementations use a Mark-and-sweep garbage collector. One of its major strengths is that it can properly deal with circular references. (The article also links this very outdated workshop paper. Don't pay much attention to it, since it is all about circular references, which should not be an issue anymore.)
   • This article goes in-depth on V8 memory management (NOTE: Node and Chrome are both based on V8).
2. If you find that memory or GC usage explodes beyond your expectation, analyze your heap memory profile to find out where memory gets allocated.
   • This SO answer explains how to do that in Chrome, but its links are outdated. This is a direct link to the relevant Chrome documentation (as of 2021).
   • For Node, I found a lot of outdated information. Currently, the easiest way to analyze your heap memory profile seems to be using the experimental --heap-prof command line argument (e.g. node --heap-prof node_modules/rollup/dist/bin/rollup -c to analyze a rollup build). Then open it in Chrome Dev Tools, via Memory -> Load.
   • Once analyzed, we can understand where/how most memory was allocated; but one crucial question has not yet been answered:
3. Given you know who the culprit (the memory hog) is, how can you find out why/where they are still lingering? And, more importantly: What is the GC root (stack pointer) of the memory hogging object?

This last question is also my question here: How can we analyze the object allocation tree in Node (or in V8 in general)? How can I find out where the objects that I identified in step (2) are kicking around?

Often, it is the answer to this question that tells us where to change our code to stop the leakage. (Of course, if your issue is memory churn, rather than memory leaks, then this question is probably not that important.)

In my example, I know that the memory is occupied by Babel AST nodes and path objects, but I don't know why they linger, that is I don't know where they are stored. If you just run Babel on its own, you can verify that it is not Babel leaking the memory. I am currently trying all kinds of tricks to find out where they are being stored, but still no luck.

Sadly, so far, I have not found any tools to help with question (3). Even relevant in-depth articles (like this and its slidedeck here) MANUALLY draw up heap allocation steps. Feels like there is no such tool, or am I wrong? If there is no tool, maybe is there a discussion about this somewhere?

Answer 1

Note that while you don't have to explicitely deallocate memory in JS, memory leaks can still arise. At the same time, Node memory profiling utilities are (almost criminally) underdocumented. Let's find out how to use them.

TLDR: Skip ahead to the hands-on section with examples below, titled "Finding Memory Leaks (with Examples)".

Memory Leaks in JS

Since JS has a GC, memory leaks only have a few possibly causes:

• You are hanging on to ("retaining") large objects, that are not used anymore, usually inside a variable in file or global scope. This is either accidental, or, part of a simplistic (indefinite) caching scheme:

  let a;
  function f() {
    a = someLargeObject;
  }
  

• Sometimes objects are lingering in retained closures. E.g.:

  let cb;
  function f() {
    const a = someLargeObject;  // `a` is retained as long as `cb`
    cb = function g() {
      eval('console.log(a)');
    };
  }
  

You can easily fix such a memory leak by either never storing to, or by manually clearing those variables. The main difficulty is to find these lingering objects.

Using Chrome Dev Tools to Profile Node Applications

Firstly, Node.js and Chrome both use the same JS engine: v8. Because of that, it was feasible for the Chrome Dev Tools team to add Node debugging and profiling support. While there are other tools available, Chrome Dev Tools (CDT) are probably more mature (and probably much better funded), which is why we will (for now) focus on how to use Chrome Dev Tools for Node memory profiling and debugging.

There are two main ways of profiling Node memory using CDT:

1. Run your app with --heap-prof to generate a heap profile log file. Then load and analyze the log in CDT.
2. Run your app with --inspect/--inspect-brk flag in order to debug your Node application in CDT. Then just use CDT's Memory tab (documentation here) to your liking.

Method 1: heap-prof

Run your app with --heap-prof to generate a heap profile log file. Then load and analyze the log in CDT.

Steps

1. Run your application with heap-prof enabled. E.g.: node --heap-prof app.js
   • You can further customize this with other heap-prof-related command line flags.
2. Look into the working directory (usually the folder from where you are running the application). There is a new file which, by default, is named Heap*.heapprofile.
3. Open a new tab in Chrome → open CDT → go to Memory tab
4. At the bottom, press Load → select Heap*.heapprofile
5. Done. You can now see where memory, still alive at the end of the recording, was allocated.

Considerations for Method 1

This step allows you to, first of all, verify a memory leak, and find out what kind of allocations or objects might be causing it.

Let's look at CDT's memory profiling tool. It has three modes:

Sadly, the log recorded by --heap-prof only contains data for mode 1. However, this mode is insufficient to answer the OP's third question: How can you find out why/where allocated objects are still lingering (that is: "retained" after not being used anymore)?

As explained in the tab: Answering that question requires the second mode.

I don't know if there is a hidden way to change the profile mode for Node, but I have not found it. I tried a few things, including adding from this list of undocumented Node.js CLI flags.

That is why @jmrk proposed method (2) in his answer:

Method 2: inspect/inspect-brk

Run your app with --inspect/--inspect-brk flag in order to debug your Node application in CDT. Then just use CDT's Memory tab (documentation here) to your liking.

Steps

1. Run application in debug mode, and halt execution at the beginning: node --inspect-brk app.js
2. Open chrome://inspect in Chrome.
3. After a few seconds, your application should show up in the list. Select it.
4. CDT are launched and you see that execution is halted at the entry point of your application.
5. Go to the Memory tab, select the 2nd mode and press the "Record" button
6. Continue execution until the memory leak was recorded. For this, either put down a breakpoint somewhere, or, if the leak persists until the end, just let the app exit naturally.
7. Go back to the Memory tab and press the "Record" button again to stop recording.
8. You can now analyze the log (see below).

Considerations for Method 2

1. Because you are now running your entire application in debug mode, everything is a lot slower.
2. Heap Mode 2 generally requires a lot more memory. If memory exceeds your Node default memory limit (about 2gb), it will just crash. Monitor your memory usage, and possibly use something like --max-old-space-size=4096 (or bigger numbers) to double the default. Or, even better, simplify your test case to use less memory and speed up profiling, if possible.
3. The "Record Allocation Stacks" option shows you the call stack of when any object was allocated. That is similar to the functionality of Profile mode 1. It is not necessary for finding memory leaks. I have not needed it so far, but if you need to map the lingering objects to their allocations, this should help.

Finding Memory Leaks (with examples)

After following the steps of Method 2, you are now looking at all information you need to find your leak.

Let's look at some basic examples:

Example 1

Code

A simplistic memory leak is examplified in the code below: file-scoped a stores data forever.

Complete Gist is here.

let a;
function test1() {
  const b = [];
  addPressure(N, b);
  a = b;
  gc(); // --expose-gc
}

test1();
debugger;

Notes:

• It is our goal to find "lingering" objects; which are "non-collectable" objects; objects that have been retained even though they are not used anymore. That is why I would usually call gc when profiling. This way we can make sure we get rid of all collectable references, and focus explicitly on the "lingering" objects.
  • You need the expose-gc flag for the gc() call; e.g.: node --inspect-brk --expose-gc app.js

Memory View

Once the breakpoint hits, I stop recording and I get this:

• The Constructor view lists all lingering objects, grouped by constructor/type.
  • Make sure, you are sorting by Shallow Size or Retained Size (both are explained here)
• We find that string is using up most memory. Let's open that up.
  • Below every Constructor, you find a list of all it's individual objects. The first (biggest) object(s) is/are often times the culprit. Select the first.
• The Retainers view now shows you where this object is still being retained.
  • Here you want to find the function that retained it for the long term (making it "linger").

Documentation on the Retainers view is not quite complete. This is how I try to navigate it until it spits out the line of code that I'm looking for:

• Select an object.
  • (Again, it's usually easiest to work through this list, sorted by size.)
• Inside the object's tree view entry: open up nested tree view entries.
• Look for anything refering to a line of code (displayed on the right-hand-side of the first column).
• Entries labeled with "context" might be more useful than others.

My findings are shown in this screenshot:

We see three functions playing a role in this object's lingering:

• The function that called gc - I'm not sure why this is. Probably related to GC internals. Might be because the gc would cache references to some (if not all) lingering objects.
• The addPressure function allocated the object. This is also where the reference that retained it came from.
• The test1 function is where we assigned the object to the file-scoped a.
  • This is the actual leak! We can fix it by either not assigning it to a, or make sure, we clear a after it's not being used anymore.

Conclusion

I hope, this helps you get started on your exciting journey to finding and eradicating your memory leaks. Feel free to ask for more information below.

Answer 2

Chrome DevTools has a "Heap Snapshot" feature, which among other things lets you inspect "retaining paths" of objects (which is, in essence, your "question 3"). See https://developers.google.com/web/tools/chrome-devtools/memory-problems/heap-snapshots for details.

You can connect DevTools to Node when you start Node with --inspect. See https://nodejs.org/en/docs/guides/debugging-getting-started/ for details.