Memory leak when emitting messages with Socket.IO + Node.js + ZMQ

Question

I have three applications talking to each other. A websocket server (1) that accepts connections from browsers, parses url to see what data is required, serves it to client if it has the data in memory, if not asks for it from another application called "fetcher" (2). Fetcher receives this job, requests it from a simple API (3) that returns JSON data, and sends it back to websocker server which publishes it to connected clients. "Fetcher" then starts to periodically check if there are updates to that url/job, and sends the new data to websocket server as they occur.

I use socket.io for client-websocket server communication. Websocket server and fetcher talks via ZMQ sockets.

I load test websocket server with 130 connections. Websocket server publishes 160KB of data to 130 clients every second. In the beginning, it uses 170mb of RAM for 130 connections, but swiftly this increases to 1GB although there are no new connections. Then socket.io's heartbeat signals starts to fail, resulting in dropped connections.

I use Nodetime to take heap snapshots. Right after 130th client connects, here's how memory looks:

346 Buffer objects with total 44MB.

In four minutes, number of Buffer objects increases dramatically (again, without new connections): there are 3012 of them with a total memory of 486MB. After 10 more minutes, there are 3535 of them with a total memory consumpion of 573MB.

I used Mozilla's memwatch to find out which line adds to memory, and found to be this function:

function notifyObservers(resourceId) {   var data = resourceData[resourceId];   io.sockets.in(resourceId).emit('data', data); } 

If I comment these lines out, memory usage stays the same so that's another confirmation.

Any ideas how this can happen? I call this function inside ZMQ's subscriber socket method and I suspect it has something to do with that.. This is the resulting code if I remove functions and merge them into one:

// Receive new resource data const resourceUpdatedSubscriber = zmq.socket('sub').connect('tcp://localhost:5433'); resourceUpdatedSubscriber.subscribe('');  resourceUpdatedSubscriber.on('message', function (data) {   var resource = JSON.parse(data);    resourceData[resource.id] = resource.data;    io.sockets.in(resourceId).emit('data', resourceData[resource.id]); }); 

All my code (including the load test) is public and you can find this web socket server here: https://github.com/denizozger/node-socketio/blob/master/server.js See line 138.

I started learning Javascript and Node.js two months ago so any comments are welcome, thanks in advance!

Answer 1

NodeJs may be use Windows Socket API ( which include memory leaks , old known bug ) https://connect.microsoft.com/VisualStudio/feedback/details/605621/winsock-c-program-causes-memory-leak

The problem is the WSACleanup will never be called until you shutdown the network services. ( Mixing up ZMq or Nodejs won't change that fact )

Now, over the time, you will lock more pages of memory and such notice an increasing after the first memory page is full.