NodeJS: Reached heap limit Allocation failed - JavaScript heap out of memory

Question

I have a VPS (Debian) with 2 GB RAM where I run pm2 in order to execute a node.js server application. I have a cache and every night, at 3 am I want to dump it into a file as a backup.

Every day at 3, I got the following error:

<--- Last few GCs --->

[717119:0x63345b0] 66921944 ms: Mark-sweep 956.0 (1043.7) -> 954.9 (1043.7) MB, 737.3 / 0.0 ms (average mu = 1.000, current mu = 0.349) allocation failure; scavenge might not succeed [717119:0x63345b0] 66922975 ms: Mark-sweep 962.8 (1043.7) -> 961.9 (1059.7) MB, 978.0 / 0.0 ms (average mu = 0.999, current mu = 0.052) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 1: 0xb6e500 node::Abort() [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 2: 0xa7e632 [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 3: 0xd47f20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 4: 0xd482c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 5: 0xf25685 [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 6: 0xf37b6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/userName/web/appName.com/public_html/appNameServer/stoc> 7: 0xf1226e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /home/userName/web/app> 8: 0xf13637 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /home/userName/web/stoc> 9: 0xef3b80 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 10: 0xeeb5f4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node /home/userName/web/stoc>11: 0xeed988 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 12: 0x131ddec v8::internal::IncrementalStringBuilder::Extend() [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 13: 0x10349d8 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 14: 0x1035c52 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>15: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>16: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>17: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>18: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>19: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>20: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>21: 0x1037f13 v8::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handle<v8::internal::JSReceiver>) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 22: 0x1039897 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/d>23: 0x103ae5f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node /hom>24: 0xdcdbf7 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 25: 0x16e9af9 [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]

I know the error is straight forward and the trivial answer would be to add more RAM, but I still want to share some more data:

1. Before the crash, the app consumed 640mb (gross), meaning the cache itself consumes less than that.
2. When I run this node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))' the result is 1662.
3. When I run the caching dump locally, it takes around 3 seconds to export the items but viewing pm2 logs on the server, it takes 14 seconds between trying to export the cache items until the server is back up again, as if the export takes much longer or maybe the crash/restart takes that much, not sure.

2022-12-27T03:00:00.016Z: exporting to json 2|appName|

2022-12-27T03:00:14.747Z: server is running on https at port...

Based on the numbers above, 640mb (gross) and the fact that there's at least one more 1gb left for node, I don't understand why there's a heap out of memory in the first place.

Please advise, thanks

Answer 1

NODE_OPTIONS (--max_old_space_size)

In my case

$ node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
4144

but

$ export NODE_OPTIONS=--max_old_space_size=8192 && node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
8240

so maybe you can modify amount of available memory using this variable.

Writing file using buffer

Probably you are writing file using

fs.writeFileSync('/tmp/test.txt', content);

but maybe you can switch to

fs.createWriteStream(path, [options])

then you will not have to store all content of file in memory in the same time.

Profiling using IDE

I solved similar cases using tools like this:

https://www.jetbrains.com/help/webstorm/v8-cpu-and-memory-profiling.html#ws_node_cpu_profiling

in console you can simply start your script with --porf flag. It will created huge, hard to read, but very detailed logs telling about memory allocation.