Cannot allocate 298930300 bytes of memory (of type "old_heap")

Question

While load testing my erlang server with increasing number (100, 200, 3000,....) of processes using +P which is the maximum number of concurrent processes, as well as making 10 process sending 1 message to the rest of the created processes, I got a message on the erlang console:

"Crash dump was written to: erl_crash.dump. eheap_alloc: Cannot allocate 298930300 bytes of memory (of type "old_heap"). Abnormal termination".

I'm using Windows XP. The is no problem when I create the process (it's working). The crash happens after the process starts communicating (sending hi & receiving hello) and this is the only problem I have (by the way, +hms which sets the default heap size of processes).

How can I resolve this?

Answer 1

Have a look at that erl_crash.dump file using the Crashdump Viewer:

/usr/local/lib/erlang/lib/observer-1.0/priv/bin/cdv erl_crash.dump

(Apologies for the Unix path; you should be able to find a cdv.bat in your installation on Windows.)

Look at the process list; in my experience there's fairly often a process with a really long message queue where you didn't expect it.

Answer 2

You ran out of memory. Try decreasing the default heap size or limit the number of processes you start.

More advanced solutions include profiling your application to see if you can save some memory there, for example better sharing of binaries or less use of lists and large messages (which will copy the data to every process it's sent to).

Answer 3

If somebody will find it useful as one of possible reasons for such problem(since I haven't found any specific answer anywhere) we've experienced similar problem with rabbitmq server (linux, 64bit, persistent queue, watermarks with default config)

eheap_alloc: Cannot allocate yyy bytes of memory (of type "heap")

eheap_alloc: Cannot allocate xxx bytes of memory (of type "old_heap")

The problem was in re-queueing too much messages at once. Our "monitoring" code used "get" message with re-queue option without limiting number of messages to get & re-queue(in our case -all messages in the queue which was 4K) So at a time it tried to add all this messages back to queue the server failed with above message.

hope it will save few hours to someone.

Answer 4

One of your processes tries allocate almost 300MB memory. You have probably memory leak in your server. In proper design you should not have so much big heap in one process if it is not intended.