Lambda AWS instant return logs

Question

I would like to have direct fast access to logs in AWS lambda.

I can query CloudWatch but It requires time to GET data.

Is it possible to get all logs of execution in Lambda and return this data in response?

---

UPDATE 15.06.2020

Thank you @John for answer, here I provide more information:

I am using dedicated logger for lambda in .NET

https://github.com/aws/aws-lambda-dotnet/tree/master/Libraries/src/Amazon.Lambda.Logging.AspNetCore

My app needs logging to debug if any problem occurs.

For example, what type of "logs" are you wanting to access

I log just some data in lambda function on different logging levels like:

1. logger.LogDebug("Lambda started");
2. logger.LogCritical(exceptions);
3. logger.LogInfo(businessDataForDebug);

I would like to get all logged statements, which where passed by logging level and return all those information in lambda response as fast as possible. Logs data must refer to particular lambda execution. Otherwise, it would be useless.

What type of data are you seeking from CloudWatch?

I need mostly logged string messages and exceptions if occur. If it is possible It could be nice to include time of particular log invocation. Moreover some "log" Id would be nice to have as reference - there is something like RequestId.

For now I can get all those logs doing query in CloudWatch but it takes time and it's too slow. I do API calls: StartQueryAsync and then GetQueryResultsAsync. Solution should be as fast as possible. I could make ugly custom logger where I can collect all logs to ex. List myLogs and simply return in response as JSON but I hope there are nice AWS solution to this purpose.

What do you mean by "logs of execution"?

I mean all logs generated by logger in Lambda simple function runtime.

LambdaFunction(request, context){
InitializeLoggerWithLogLevel(debug);
logger.LogInfo("Log Info"); //Not logged due to min log level Debug
logger.LogTrace("Log Trace"); //Logged, trace > debug
logger.LogDebug("Log debug"); //Logged, debug == debug

Are you referring to the output of AWS Lambda functions that are stored in Amazon CloudWatch Logs?

Yes, I can query logs from there but it takes a few seconds to get it. I would have faster fetching data.

I can believe that AWS doen't provide such solution. Workaround could be to delegate getting logs to another lambda by ex. SQS or move logs of executed particular lambda function to S3 bucket and then give user link to download. My first main lambda must be fast, processing many requests like machine gun ;).

Please advice me the best approach. I am open for further discussion.

Regards,

Answer 1

If you're hitting 50 transactions per second the CloudWatch transactions get throttled. You can ask for an increase on this limit: https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-cloudwatch-logs

Ref: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html

If it is possible It could be nice to include time of particular log invocation. Moreover some "log" Id would be nice to have as reference - there is something like RequestId. For now I can get all those logs doing query in CloudWatch but it takes time and it's too slow.

If you're after a FAST solution and you really are pushing the CloudWatch limits use DynamoDB.