NodeJS stream out of AWS Lambda function

Question

We are trying to migrate our zip microservice from regular application in nodejs Express to AWS API Gateway integrated with AWS Lambda.
Our current application sends request to our API, gets list of attachments and then visits those attachments and pipes their content back to user in form of zip archive. It looks something like this:

module.exports = function requestHandler(req, res) {

  //...
  //irrelevant code
  //...

  return getFileList(params, token).then(function(fileList) {
    const filename = `attachments_${params.id}`;
    res.set('Content-Disposition', `attachment; filename=${filename}.zip`);

    streamFiles(fileList, filename).pipe(res); <-- here magic happens
  }, function(error) {
    errors[error](req, res);
  });
};

I have managed to do everything except the part where I have to stream content out of Lambda function.
I think one of possible solutions is to use aws-serverless-express, but I'd like a more elegant solution.

Anyone has any ideas? Is it even possible to stream out of Lambda?

Answer 1

Unfortunately lambda does not support streams as events or return values. (It's hard to find it mentioned explicitly in the documentation, except by noting how invocation and contexts/callbacks are described in the working documentation).

In the case of your example, you will have to await streamFiles and then return the completed result.

(aws-serverless-express would not help here, if you check the code they wait for your pipe to finish before returning: https://github.com/awslabs/aws-serverless-express/blob/master/src/index.js#L68)

n.b. There's a nuance here that a lot of the language SDK's support streaming for requests/responses, however this means connecting to the stream transport, e.g. the stream downloading the complete response from the lambda, not listening to a stream emitted from the lambda.