Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

NodeJS/ExpressJS send response of large amount of data in 1 stream

I'm prototyping an app using the native mongo rest api where Node returns about 400K of json. I use the following to maket he request to mongo's native api and return the result:

http.request(options, function(req)
  {
    req.on('data', function(data)
      {
console.log(data,data.rows);
        response.send( 200, data );
      }
    );
  }
)
.on('error', function(error)
  {
console.log('error\t',error);
    response.send(500, error);
  }
)
.end();

When I hit http://localhost:8001/api/testdata via curl, the response is proper (both what is outputted to Node's console from the console.log and what is received by curl). But when I hit it via ajax in my app, the stream is…interupted, even data outputted to Node's console (Terminal) is odd: It has multiple EOFs, and the Network > response for the call in chrome's dev tools ends at the first EOF.

One other strange thing: data looks like:

{
    "offset": 0,
    "rows": [ … ]
}

but in neither Node nor client-side (angular) can I reference data.rows (it returns undefined). typeof data returns [object Object].

EDIT The request headers for both curl and angular (as reported by Node) are:

req.headers: {
  'x-action': '',
  'x-ns': 'test.headends',
  'content-type': 'text/plain;charset=utf-8',
  connection: 'close',
  'content-length': '419585'
}

EDIT I checked response headers in both angular and curl directly (instead of from Node), annnd there's a disagreement (same output from both curl and angular directly instead of from node):

access-control-allow-headers: "Origin, X-Requested-With, Content-Type, Accept"
access-control-allow-methods: "OPTIONS,GET,POST,PUT,DELETE"
access-control-allow-origin: "*"
connection: "keep-alive"
content-length: "65401" // <---------------- too small!
content-type: "application/octet-stream"
//             ^-- if i force "application/json"
// with response.json() instead of response.send() in Node,
// the client displays octets (and it takes 8s instead of 0s)
date: "Mon, 15 Jul 2013 18:36:50 GMT"
etag: ""-207110537""
x-powered-by: "Express"
like image 576
Jakob Jingleheimer Avatar asked Jul 12 '13 19:07

Jakob Jingleheimer


People also ask

How much traffic can express js handle?

There's a benchmark made by Fastify creators, it shows that express. js can handle ~15K requests per second, and the vanilla HTTP module can handle 70K rps.

Can we send multiple response in node JS?

You can only send one HTTP response for one HTTP request. However, you can certainly write whatever kind of data in the response that you want. That could be newline-delimited JSON, multipart parts, or whatever other format you choose.

How do you handle a large data set in node JS?

To do that we create a request of type stream using axios. In case the requested file has the . zip extension, we are going to pipe the response through unzip, otherwise we write the stream to the disk using the node. js native fs.


1 Answers

Node's http.request() returns data in chunks for streaming (would be nice if they explicitly state this). Thus it's necessary to write each chunk to the body of Express's response, listen for the end of the http request (which is not really documented), and then call response.end() to actually finish the response.

var req = http.request(options, function(res)
  {
    res.on( 'data', function(chunk) { response.write(chunk); } );
    res.on( 'end', function() { response.end(); } );
  }
);
req.on('error', function(error) { … });
req.end();

Where response is Express's response the the initial client request (curl or angular's ajax call).

like image 57
Jakob Jingleheimer Avatar answered Oct 05 '22 10:10

Jakob Jingleheimer