Splash memory limit (scrapy)

Question

I have splash started from docker. I create big lua script for splash and scrapy, and then it's run i see problem:

Lua error: error in __gc metamethod (/app/splash/lua_modules/sandbox.lua:189: script uses too much memory

How i can encrease memory for splash?

Answer 1

Unfortunately, as of Splash 2.3.2, there is no a built-in way to raise these limits. Limit is hardcoded here: https://github.com/scrapinghub/splash/blob/7b6612847984fc574ebbedf9c3c750180cd93813/splash/lua_modules/sandbox.lua#L176 - you can change the value, and then rebuild Docker image by running docker build -t splash . from Splash source checkout, and then use this image instead of image from DockerHub.

Answer 2

I solve my problem by optimizing the lua script. It turns out splash:select("a#story-title").node.innerHTML is much heavier than splash:evaljs('document.getElementById("story-title").innerHTML;')