invoke lambda function with hours delay

Question

I am trying to figure out what is the best approach to invoke an aws lambda function 5 hours later. I have another lambda function that would issue multiple retrieval jobs to grab items out of aws glacier, and I need a solution to run another lambda function on each of the items once they are retrieved, which is about 5 hours. I was thinking about using sns but was wondering if there are other approaches to this. any input is appreciated.

Answer 1

Besides using CloudWatch, another interesting approach in your case would be to use AWS Step Functions:

1. Either use a wait state by setting a fixed period (or even a dynamic one if you provide input data to the state machine):

   {
     "Comment": "An example of the Amazon States Language using wait states",
     "StartAt": "WaitState",
     "States": {
       "WaitState": {
         "Type": "Wait",
         "Seconds": 10,
         "Next": "MyLambda"
       },
       "MyLambda": {
         "Type": "Task",
         "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
         "End": true
       }
     }
   }
   

2. Or you can use a separate Lambda function in a task state combined with a choice state which checks in a loop if the other function should run:

   {
     "Comment": "A state machine that submits a Job to AWS Batch and monitors the Job until it completes.",
     "StartAt": "Wait X Seconds",
     "States": {
       "Wait X Seconds": {
         "Type": "Wait",
         "SecondsPath": "$.wait_time",
         "Next": "Get Job Status"
       },
       "Get Job Status": {
         "Type": "Task",
         "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:CheckJob",
         "Next": "Job Complete?"
       },
       "Job Complete?": {
         "Type": "Choice",
         "Choices": [
           {
             "Variable": "$.status",
             "StringEquals": "RUNNING",
             "Next": "Wait X Seconds"
           },
           {
             "Variable": "$.status",
             "StringEquals": "SUCCEEDED",
             "Next": "Do Job"
           }
         ],
         "Default": "Wait X Seconds"
       },
       "Do Job": {
         "Type": "Task",
         "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:DoJob",
         "End": true
       }
     } 
   }
   

Answer 2

I feel that the cloudwatch and step-functions based solutions do not scale well and they are pretty heavy code-wise.

I recommend creating two SQS queues. One will be the waiting queue, and the other one will be the execution queue.

Waiting Queue: Set the execution queue as the deadletter queue of the waiting queue. Set the Default Visibility Timeout equal to the time you want to wait to run the lambda (up to 12 hours). Set maximum receives of 1. Create a Lambda that consumes an SQS message and returns an error. Add the waiting queue as a trigger to this lambda.

exports.handler =  async function(event, context) {
  throw new Error('Go back to the waiting queue and wait');
}

Execution Queue: Add the execution queue as a trigger to your Lambda (that you want to schedule).

Then, to schedule your lambda you need to simply post a message to the waiting queue.

The message will go to the waiting queue where it will be consumed by the error-out lambda. It will tgen sit on the waiting queue until its visibility times out and it will be pushed to the deadletter execution queue for pickup by your scheduled lambda.