"LAMBDA_RUNTIME" Error on high-volume Lambda Function

Question

I'm currently using a Lambda Function written in Javascript that is setup with an SQS event source to automatically pull messages from an SQS Queue and do some basic processing on the message contents. I cannot show the code but the summary of the lambda function's execution is basically:

For each message in the batch it receives as part of the event:

1. It parses the body, which is a JSON string, into a Javascript object.
2. It reads an object from S3 that is listed in the object using getObject.
3. It puts a record into a DynamoDB table using put.
4. If there were no errors, it deletes the individual SQS message that was processed from the Queue using deleteMessage.

This SQS queue is high-volume and receives messages in-bulk, regularly building up a backlog of millions of messages. The Lambda is normally able to scale to process hundreds of thousands of messages concurrently. This solution has worked well for me with other applications in the past but I'm now encountering the following intermittent error that reliably begins to appear as the Lambda scales up:

[ERROR] [#############] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 400.

I've been unable to find any information anywhere about what this error means and what causes it. There appears to be not discernible pattern as to which executions encounter it. The function is usually able to run for a brief period without encountering the error and scale to expected levels. But then, as you can see, the error starts to appear quite suddenly and completely destroys the Lambda throughput by forcing it to auto-scale down:

Does anyone know what this "LAMBDA_RUNTIME" error means and what might cause it? My Lambda Function runtime is Node v12.

Answer 1

I have this error only that I get:
[ERROR] [1638918279694] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413.

I went to the lambda function on aws console and ran the test with a custom event I build and the error I got there was:

{
    "errorMessage": "Response payload size exceeded maximum allowed payload size (6291556 bytes).",
    "errorType": "Function.ResponseSizeTooLarge"
}

So this is the actual error that cloudwatch doesn't return but the testing section of the lambda function console do.

I think I'll have to return info to an S3 file or something, but that's another matter.