Amazon AWS Lambda NodeJS logging

Question

I have a sync job (in Node.js) which has to process several hundred of documents in one batch. For each of them also perform several tasks. As usually, after deployment, such job will become a blackbox: without propper logging it is impossible to find a problem.

Therefore, I log any reasonable information - which document job is being processed, what task is performing now etc. I use console.log / console.error for logging. This results in a very large log file, which is not that big problem when running localy.

Once deployed on AWS, is there any best practice / limitation for logging? I am afraid of costs also.

Thanks!

Answer 1

There are many ways to approach this.

If your main need is visualizing your data workflow, using AWS Step Functions would do it. They recently launched a cheap version called Express Workflows.

Break down into multiple Lambdas, one for each task. Step Functions will take care of all the orchestration, error handling, retrying, etc. Using AWS SQS may also be beneficial to smooth out the batching process.

For full visibility, a specialized tool will be required. In serverless, we don't have control over the infrastructure, so a different approach is required. I'd recommend to check out Dashbird.