How to wait for async actions inside AWS Lambda?

Question

I am trying to process uploaded file in S3. Since getObject is asyncronous main function ends before processing is done, and AWS kills lambda in 3-4 seconds.

Even worse, processing method also has async operations in it - it makes http calls.

On high level, my code looks like:

exports.handler = function(event, context) {     // Get the object from the event and show its content type     var bucket = event.Records[0].s3.bucket.name;     var key = event.Records[0].s3.object.key;     var params = {         Bucket: bucket,         Key: key     };     s3.getObject(params, function(err, data) {         if (err) {              ...         } else {             processFile(data.Body.toString(), 0);             console.log("ok");         }     });     //need to wait here till processFile is done };  processFile = function(content, start) {   ... build url to call   http.get(url, function(res) {       console.log("Got response: " + res.statusCode + ");     processFile(content, start + 1);   }); } 

I find out that there is async in nodejs but it is not included by amazon; Both require('async') or require('sleep) causes errors.

Lambda timeout configured to 60 seconds, but it exits in 3-4 seconds.

Answer 1

The life of a dev is constantly changing and we now have NodeJS 8 on lambda. For anyone looking at this now check out:

Lambda node 8.10 vs node 6.10 comparison: https://aws.amazon.com/blogs/compute/node-js-8-10-runtime-now-available-in-aws-lambda/

Basics of JS async: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

Even more aws sdk examples: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-promises.html

Details on wtf the .promise() method is in the first link: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Request.html#promise-property

Here is my take at a basic example (try pasting into your own lambda):

exports.handler = async (event) => {          function wait(){          return new Promise((resolve, reject) => {              setTimeout(() => resolve("hello"), 2000)          });      }            console.log(await wait());      console.log(await wait());      console.log(await wait());      console.log(await wait());      console.log(await wait());      console.log(await wait());            return 'exiting'  };

The above yields:

As you can see it waited 12 seconds without killing my function :)

TODO more than one thing per await, use Promise.all([]) syntax like this:

exports.handler = async (event) => {     var uploadPromises = [];     folder.files.forEach(file => {         uploadPromises.push( s3.putObject({             Bucket: "mybucket",             Key: file.name,             Body: file.data         }).promise());     });      await Promise.all(uploadPromises);     return 'exiting' };  

Orignal Answer Below

I had the exact same issue on my hands.

The problem is the javascript event loop is empty so Lambda thinks it's done.

This is how I solved this problem. I realize this is not ideal, and I wish there was a better way, but I didn't want to a) add libraries, b) coordinate lambda invocations, or c) switch to another language.

At the end of the day it works.

    exports.handler = (event, context, callback) => {          var response;          var callBackCount;            /*          Ensures the javascript event loop is never empty.          This is the key to keeping lambda from exiting early          */          setInterval(function(){}, 1000);            /*          Tell lambda to stop when I issue the callback.          This is super important or the lambda funciton will always go until it hits the timeout limit you set.          */          context.callbackWaitsForEmptyEventLoop = false;                    //My way of determining when I'm done with all calls          callBackCount = 0;                  //My info to return          response = "";                    //Various functions that make rest calls and wait for a response          asyncFunction1();          asyncFunction2();          asyncFunction3();            //Same for asyncFunction 2 and 3          function asyncFunction1(){            response += callBackResponseForThisMethod;                    returnResponse();          }            function returnReponse(){              callBackCount++;                if(callBackCount == 3){                //Lambda will stop after this as long as    context.callbackWaitsForEmptyEventLoop was set to false                 callback(null, JSON.stringify(response));              }          }        };

http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html

Answer 2

Using async/await

let AWS = require('aws-sdk'); let lambda = new AWS.Lambda(); let data; exports.handler = async (event) => {     try {         data = await lambda.getAccountSettings().promise();     }     catch (err) {         console.log(err);         return err;     }     return data; };