AWS Lambda and Apache Airflow integration

Question

wondered if anyone could shed some light on this issue:

I'm trying to locate the Airflow REST API URL to initiate a DAG to Run from AWS Lambda Function.

So far from looking at all the relevant documentation provided from the Apache Incubator Site, the only guidance to solved the problem is by using this URL structure in the Lambda (python 3.6) Code:

Apache Experimental API: https://airflow.apache.org/api.html#endpoints

based on that link the syntax should read:

http://airflow_hostname/api/experimental/dags/<DAG_ID>/

However this fails to initiate the Specific DAG in from my AWS Lambda Function, any help/guidance or pointers would be great.

Answer 1

In the Airflow 1.10 the following post request will work:

curl -X POST http://localhost:8080/api/experimental/dags/<dag_id>/dag_runs -H 'Cache-Control: no-cache' -H 'Content-Type: application/json'    -d '{"conf":"{\"key\":\"value\"}"}'

You need to pass an empty JSON string otherwise the post request will fail.

Answer 2

Just a followup here...this post helped me as well. The current answer mentions that a JSON string (empty or not) is required for the POST body...which is correct. The other point which is subtle, is that the url in the OP is missing the "/dag_runs" ending portion of the url.

For clarify, here is a simple AWS lambda function that invokes an Airflow DAG named "hello_world":

import os
from botocore.vendored import requests
import json

def lambda_handler(event, context):
    print("Event Passed to Handler: " + json.dumps(event))
    data = {}
    url = 'http://ec2-11-111-11-111.us-east-2.compute.amazonaws.com:8080/api/experimental/dags/hello_world/dag_runs'
    print('sending POST request: ' + url)
    r = requests.post(url, json.dumps(data))
    print('response:')
    print(r)
    return r.status_code