Prevent node from running out of memory in a docker build that a 600 MB memory limit

Question

I am running docker build with a limit on the build's memory and CPU. To stay within the build's CPU and memory limits, I am also limiting Node to a heap size of 325 MB. This is the docker build command.

docker build --build-arg NODE_OPTIONS=--max-old-space-size=325 \
             --memory=600m --memory-swap=-1 \ 
             --cpu-period=100000 --cpu-quota=50000 \
             --no-cache --tag farm_app_image:latest --file Dockerfile .

Build Resource Summary

• Node JS Heap Limit: 325 MB
• Build Memory: 600 MB with unlimited use of swap files.
• Build CPU Time: 50%

Despite having a Node heap limit that is below the build memory, and despite having unlimited swap, the npm run build runs out of memory on the react-scripts build step.

Error Ouput

  > react-scripts build                            

  Creating an optimized production build...                                                                                                                                          
EXEC : FATAL error : Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory [/app/FarmLandLasanga/FarmLandLasanga.csproj]                       

  <--- Last few GCs --->                                                                                                                                                             

  [186:0x5e6ca40]   143688 ms: Mark-sweep 319.3 (329.2) -> 319.2 (329.9) MB, 1278.3 / 0.0 ms  (average mu = 0.118, current mu = 0.002) allocation failure scavenge might not succeed 
  [186:0x5e6ca40]   145181 ms: Mark-sweep 320.3 (329.9) -> 320.2 (331.2) MB, 1488.0 / 0.1 ms  (average mu = 0.060, current mu = 0.003) allocation failure scavenge might not succeed 


  <--- JS stacktrace --->                                                                                                                                                            

  ==== JS stack trace =========================================                                                                                                                      

      0: ExitFrame [pc: 0x1391439]                                                                                                                                                   
      1: StubFrame [pc: 0x1316d29]                                                                                                                                                   
  Security context: 0x154b83ac08d1 <JSObject>                                                                                                                                        
      2: /* anonymous */ [0x14bbf66e0a09] [/app/FarmLandLasanga/ClientAppTypeScript/node_modules/webpack-sources/node_modules/source-map/lib/source-node.js:~86] [pc=0xa4979a73d50](t
his=0x1e34cc96a351 <JSFunction SourceNode (sfi = 0xd8bc3128ee1)>,0x307dd6e664f1 <Object map = 0xb3550d64299>)                                                                        
      3: arguments adaptor frame: 3->1                                                                                                                                               
      4:...                                                                                                                                                                          


  Writing Node.js report to file: report.20191116.195427.186.0.001.json                                                                                                              
  Node.js report completed                                                                                                                                                           
   1: 0x9e9f40 node::Abort() [node]                                                                                                                                                  
   2: 0x9ec192 node::OnFatalError(char const*, char const*) [node]                                                                                                                   
   3: 0xb4611e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]                                                                                         
   4: 0xb46499 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]                                                                           
   5: 0xcf3535  [node]                                                                                                                                                               
   6: 0xcf3bc6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]                                                                                            
   7: 0xd003fa v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]                                                              
   8: 0xd01305 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]                                  
   9: 0xd03dac v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]           
  10: 0xcca66b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]                                                
  11: 0x100eb9e v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]                                                                  
  12: 0x1391439  [node]                                                                                                                                                              
  npm ERR! code ELIFECYCLE                                                                                                                                                           
  npm ERR! errno 1                                                                                                                                                                   
  npm ERR! [email protected] build: `react-scripts --max_old_space_size=325 build`                                                                                             
  npm ERR! Exit status 1                                                                                                                                                             
  npm ERR!                                                                                                                                                                           
  npm ERR! Failed at the [email protected] build script.                                                                                                                       
  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.                                                                                 

  npm ERR! A complete log of this run can be found in:                                                                                                                               
  npm ERR!     /root/.npm/_logs/2019-11-16T19_54_27_392Z-debug.log                                                                                                                   

Question

What other changes, if any, can I make to the docker build command so that it succeeds with a 600 MB limit?

References

• https://docs.docker.com/config/containers/resource_constraints/
• https://erikcorry.blogspot.com/2012/11/memory-management-flags-in-v8.html

Additional Attempts

• With --memory=900m the build succeeds.
• react-scripts --max_old_space_size=325 build inside of Docker fails.
• react-scripts --max_old_space_size=325 build outside of Docker succeeds.

Answer 1

Can you change your base image? I think that can help. This memory leak problem was in NodeJS v10 and v12, just use latest LTS version (v14) and also alpine version of the image.

Answer 2

when using a GC language like JS or Java you have the most popular part of the language memory component which is the heap but you also have another part that is being used when running or building an application, that is the stack.

In any way, I would suggest monitoring the docker build memory usage with docker stats or any other monitoring tool you prefer as well as the react-scripts build memory usage if you use the --expose-gc option (in reference to this thread) and check the docker build logs for any spikes in usage. Running the react-scripts build command outside could help to try and determine what memory requirements the build process actually requires and setting it to it plus some buffer in the container after.

It could be that you're limiting your heap size for nothing to half of the container total memory capacity you've set (600MB), try increasing the heap memory capacity while keeping the containers one the same (increase --max_old_space_size while keeping --memory at 600MB).

Anbother option that seems to help in this and that case is to upgrade node base image to node:lts or node:14.16.0 (are the same image as of the time of writing this).

Answer 3

I just write a code, to get the total memory on a linux machine, deduct 400mb (to other server resources), and assign it as node memory limit.

  FREE=$(free -m);
  echo "FREE: ${FREE}"
  FREE_FINAL=$(echo "$FREE" | grep -F Mem:  | grep -o "[0-9]*" | grep -o -m1 "^[0-9]*")
  echo "FREE_FINAL: ${FREE_FINAL}"
  MEM_LIMIT=$(($FREE_FINAL-400))
  echo "MEM_LIMIT: ${MEM_LIMIT}"
  echo " - "

  export MEMORY_LIMIT="${MEM_LIMIT:=3500}";

  export NODE_OPTIONS="--max-old-space-size=${MEMORY_LIMIT}"
  echo "NODE_OPTIONS: ${NODE_OPTIONS}"

You can add it to a entrypoint.sh file (that will run on build time, but on each docker start as well), or on a build.sh file (that will get this value to pass to your docker build command as argument.