Rate-limiting a Worker for a Queue (e.g.: SQS)

Question

Every day, I will have a CRON task run which populates an SQS queue with a number of tasks which needs to be achieved. So (for example) at 9AM every morning, and empty queue will receive ~100 messages that will need to be processed.

I would like a new worker to be spun up every second until the queue is empty. If any task fails, it's put at the back of the queue to be re-run.

For example, if each task takes up to 1.5 seconds to complete:

• after 1 second, 1 worker will have started message A
• after 2 seconds, 1 worker may still be running message A and 1 worker will have started running message B
• after 100 seconds, 1 worker may still be running message XX and 1 worker will pick up message B because it failed previous
• after 101 seconds, no more workers are propagated until the next morning

Is there any way to have this type of infrastructure configured within AWS lambda?

Answer 1

One way, though I'm not convinced it's optimal:

A lambda that's triggered by an CloudWatch Event (say every second, or every 10 seconds, depending on your rate limit). Which polls SQS to receive (at most) N messages, it then "fans-out" to another Lambda function with each message.

---

Some pseudo code:

# Lambda 1 (schedule by CloudWatch Event / e.g. CRON)
def handle_cron(event, context):
    # in order to get more messages, we might have to receive several times (loop)
    for message in queue.receive_messages(MaxNumberOfMessages=10):
        # Note: the Event InvocationType so we don't want to wait for the response!
        lambda_client.invoke(FunctionName="foo", Payload=message.body, InvocationType='Event')

and

# Lambda 2 (triggered only by the invoke in Lambda 1)
def handle_message(event, context):
    # handle message
    pass