Need to run a aws lambda function which takes more than 15 minutes to complete?

Question

My Lambda function has limit 15 minutes which was 5 minutes ealier.Lambda process is automatically terminated after 15 minutes but my process takes more than 15 minutes. How I can manage ?

Answer 1

Typically use a Fat Lambda strategy or Step Function

Fat Lambda Strategy

A Fat Lambda strategy is used when your task is singular but has a long-running execution time and/or you have heavy hardware requirements. The idea is that you would create a script that executes your long process and put it into a docker container that's hosted in Fargate. Meaning no limits to execution time and access to powerful hardware (How to create a Fat Lambda https://youtu.be/XUp9SHIHU8w)

Step Function strategy

A Step Function strategy is used to break down your entire process into smaller steps. Usually, a step function strategy would work for you if your process could have lots of miniature stages linked together instead of a big colossal job attempting to do everything simultaneously. Bear in mind that a "Fat Lambda" can also be triggered by a Step Function (How to create a Step Function https://www.youtube.com/watch?v=s0XFX3WHg0w)

Also, another note, remember lambdas can also trigger other lambdas. So you might even be able to have different lambdas run bits of your lambda code. For example, a FOR loop sends off a lot of mini lambdas to run small tasks. You might not even need a Step Function or a Fat Lambda.

If you're stuck on what to choose, follow the below. It will help you reason with your problem.

Singular Lambda >> Lambda invoke another Lambda? >> Step Functions? >> Fargate (Fat Lambda)?