How to wait for a step completion in AWS EMR cluster using Boto3

Question

Given a step id I want to wait for that AWS EMR step to finish. How can I achieve this? Is there a built-in function?

At the time of writing, the Boto3 Waiters for EMR allow to wait for Cluster Running and Cluster Termination events:

EMR Waiters

Answer 1

There is now a waiter available for step complete events. It was added in a recent boto3 version.

http://boto3.readthedocs.io/en/latest/reference/services/emr.html#EMR.Waiter.StepComplete

Example code:

import boto3

client = boto3.client("emr")
waiter = client.get_waiter("step_complete")
waiter.wait(
    ClusterId='the-cluster-id',
    StepId='the-step-id',
    WaiterConfig={
        "Delay": 30,
        "MaxAttempts": 10
    }
)

Answer 2

There is no built-in function in Boto3. But you can write your own waiter.

See: describe_step

Call describe_step with cluster_id and step_id. The response is a dictionary that contains detail about the step. One of the keys is 'State' that has information about the step state. If the state is not COMPLETED, wait for few seconds try again until it is COMPLETED or the wait time exceeds your limit.

'State': 'PENDING'|'CANCEL_PENDING'|'RUNNING'|'COMPLETED'|'CANCELLED'|'FAILED'|'INTERRUPTED'