AWS Lambda start an async process with Python (return without waiting)

Question

I should start by saying I really know nothing about AWS, Lambda, or for that matter much about Python, so the likelihood of dumb mistakes is high.

I'm trying to create a system wherein a user POSTS a thing, the handler quickly passes them back a jobID, and then the user can revisit later to check on the process/get the result.

Basically:

def long_function(jobID, stuff): //dont know how to define this to be called asychronously
  //slow-running code
  //stuff the result in dynamoDB
  return

def lambda_handler(event, context):
    jobID = uuid.uuid4.hex
    stuff = event['stuff']
    long_function(jobID, stuff) //dont know how to write this call
    return jobID //should run before long_function completes

My understanding is that general python async stuff (which I'm also unfamiliar with) won't work, and I need to do a particular Lambda invocation. I'm failing to find a clear and simple way to do this (especially with python syntax)

Answer 1

You are correct, the lambda function, once it returns, is done, and there is no way around that.

What you can do, also as you predicted, is launch another lambda function from within the first function (or a Batch job, or make a call to an EC2 instance for that matter).

You will need boto3 (it comes by default in all lambda Python environments) and you will need to give the lambda function the necessary IAM permissions to invoke another lambda function.

You can call either the original lambda function or another function. The code below assumes it is calling the same function.

The code itself is fairly simple:

import json
import boto3

def long_function(jobID, stuff):
    """Do something that takes a long time."""
    ...

def lambda_handler(event, context):
    # Unpack the event
    stuff = event["stuff"]
    is_long_func = event.get("is_long_func", False)

    # Route to the desired action.
    if not is_long_func:
        # Create a new job ID
        jobID = uuid.uuid4.hex

        # This is where the real business happens: invoke this
        # same lambda function with a different event.
        lam = boto3.client('lambda')
        lam.invoke_async(
            FunctionName="this-lambda-function-name",
            InvokeArgs=json.dumps({
                'jobID': jobID,
                'stuff': stuff,
                'is_long_func': True
            }).encode('utf-8')
        )
        return jobID

    else:
        # Run the long function.
        jobID = event["jobID"]
        long_function(jobID, stuff)
        return "OK"

If you need the result of the function, you can save the result of the function to s3 or similar and retrieve the contents with additional calls to the lambda function.