How to scale Gatsby to 1000s of static files without running out of memory?

Question

I'm developing a React web app with Gatsby. After adding over a 1000 static markdown files, the build process requires a lot of memory. I'm running the builds under AWS Lambda, and I've started to get SIGKILLs once the memory limit of 1536MB is exhausted.

How could Gatsby builds be scaled to thousands of files without requiring more and more memory? According to the developer, it should be possible.

Answer 1

Gatsby pulls all site content, css, js into memory while doing a build. Potentially there's ways to reduce the amount of memory Gatsby uses but currently the best way to reduce your memory usage would be to split the site into identical but separate sites. I.e. keep the templates/css the same but 1000s of Markdown files into multiple sites and build each separately in sequence.

Answer 2

node --max-old-space-size=8192 node_modules/.bin/gatsby build

That's how I ran the benchmark when I was trying to make webpack faster. Was building upwards of 5000+ markdown posts.

Answer 3

Try disabling the BABEL_CACHE environment variable. I have run into problems on AWS lambda, too, and moved to Docker based builds.