NodeJS: What is the proper way to handling TCP socket streams ? Which delimiter should I use?

Question

From what I understood here, "V8 has a generational garbage collector. Moves objects aound randomly. Node can’t get a pointer to raw string data to write to socket." so I shouldn't store data that comes from a TCP stream in a string, specially if that string becomes bigger than Math.pow(2,16) bytes. (hope I'm right till now..)

What is then the best way to handle all the data that's comming from a TCP socket ? So far I've been trying to use _:_:_ as a delimiter because I think it's somehow unique and won't mess around other things.

A sample of the data that would come would be something_:_:_maybe a large text_:_:_ maybe tons of lines_:_:_more and more data

This is what I tried to do:

net = require('net');
var server = net.createServer(function (socket) {
    socket.on('connect',function() {
        console.log('someone connected');
        buf = new Buffer(Math.pow(2,16));  //new buffer with size 2^16
        socket.on('data',function(data) {
            if (data.toString().search('_:_:_') === -1) {    // If there's no separator in the data that just arrived...
                buf.write(data.toString());   // ... write it on the buffer. it's part of another message that will come.
            } else {        // if there is a separator in the data that arrived
                parts = data.toString().split('_:_:_'); // the first part is the end of a previous message, the last part is the start of a message to be completed in the future. Parts between separators are independent messages
                if (parts.length == 2) {
                    msg = buf.toString('utf-8',0,4) + parts[0];
                    console.log('MSG: '+ msg);
                    buf = (new Buffer(Math.pow(2,16))).write(parts[1]);
                } else {
                    msg = buf.toString() + parts[0];
                    for (var i = 1; i <= parts.length -1; i++) {
                        if (i !== parts.length-1) {
                            msg = parts[i];
                            console.log('MSG: '+msg);
                        } else {
                            buf.write(parts[i]);
                        }
                    }
                }
            }
        });
    });
});

server.listen(9999);

Whenever I try to console.log('MSG' + msg), it will print out the whole buffer, so it's useless to see if something worked.

How can I handle this data the proper way ? Would the lazy module work, even if this data is not line oriented ? Is there some other module to handle streams that are not line oriented ?

Answer 1

It has indeed been said that there's extra work going on because Node has to take that buffer and then push it into v8/cast it to a string. However, doing a toString() on the buffer isn't any better. There's no good solution to this right now, as far as I know, especially if your end goal is to get a string and fool around with it. Its one of the things Ryan mentioned @ nodeconf as an area where work needs to be done.

As for delimiter, you can choose whatever you want. A lot of binary protocols choose to include a fixed header, such that you can put things in a normal structure, which a lot of times includes a length. In this way, you slice apart a known header and get information about the rest of the data without having to iterate over the entire buffer. With a scheme like that, one can use a tool like:

• node-buffer - https://github.com/substack/node-binary
• node-ctype - https://github.com/rmustacc/node-ctype

As an aside, buffers can be accessed via array syntax, and they can also be sliced apart with .slice().

Lastly, check here: https://github.com/joyent/node/wiki/modules -- find a module that parses a simple tcp protocol and seems to do it well, and read some code.

Answer 2

You should use the new stream2 api. http://nodejs.org/api/stream.html

Here are some very useful examples: https://github.com/substack/stream-handbook

https://github.com/lvgithub/stick