Coping with, and minimizing, memory usage in Common Lisp (SBCL)

Question

I have a VPS with not very much memory (256Mb) which I am trying to use for Common Lisp development with SBCL+Hunchentoot to write some simple web-apps. A large amount of memory appears to be getting used without doing anything particularly complex, and after a while of serving pages it runs out of memory and either goes crazy using all swap or (if there is no swap) just dies.

So I need help to:

• Find out what is using all the memory (if it's libraries or me, especially)
• Limit the amount of memory which SBCL is allowed to use, to avoid massive quantities of swapping
• Handle things cleanly when memory runs out, rather than crashing (since it's a web-app I want it to carry on and try to clean up).

I assume the first two are reasonably straightforward, but is the third even possible? How do people handle out-of-memory or constrained memory conditions in Lisp?

(Also, I note that a 64-bit SBCL appears to use literally twice as much memory as 32-bit. Is this expected? I can run a 32-bit version if it will save a lot of memory)

Answer 1

To limit the memory usage of SBCL, use --dynamic-space-size option (e.g.,sbcl --dynamic-space-size 128 will limit memory usage to 128M).

To find out who is using memory, you may call (room) (the function that tells how much memory is being used) at different times: at startup, after all libraries are loaded and then during work (of cource, call (sb-ext:gc :full t) before room not to measure the garbage that has not yet been collected).

Also, it is possible to use SBCL Profiler to measure memory allocation.

Answer 2

Find out what is using all the memory (if it's libraries or me, especially)

Attila Lendvai has some SBCL-specific code to find out where an allocated objects comes from. Refer to http://article.gmane.org/gmane.lisp.steel-bank.devel/12903 and write him a private mail if needed.

Be sure to try another implementation, preferably with a precise GC (like Clozure CL) to ensure it's not an implementation-specific leak.

Limit the amount of memory which SBCL is allowed to use, to avoid massive quantities of swapping

Already answered by others.

Handle things cleanly when memory runs out, rather than crashing (since it's a web-app I want it to carry on and try to clean up).

256MB is tight, but anyway: schedule a recurring (maybe 1s) timed thread that checks the remaining free space. If the free space is less than X then use exec() to replace the current SBCL process image with a new one.