How to trigger cloud composer DAG using cloud function when a file is added to cloud storage using Python 3.7

Question

Every time a document is placed in a certain bucket I want to start a DAG workflow to analyse this document. I need to trigger the DAG work flow using cloud functions using Cloud storage trigger and event type Finalize and create

Answer 1

Update - as of September 2019, there now exists a Python example in the docs, and the snippets in the docs can be found on GitHub.

---

Take a look at this doc, which shows you how to Trigger a DAG using Google Cloud Functions and a bucket of your choosing on a Finalize/Create event

[edited to add Python specific details]

Since you need Python, and this doc is for NodeJS, you'll have to do the adaptation on your own for now. Calls to Airflow's API in Composer go through the identity aware proxy (see this diagram, which is also shown below, for more detail), and so your post request to trigger the DAG will also have to go through the identity aware proxy.

Luckily, there is a great Python example for making a post request to IAP in the python-docs-samples sample repo. If you copy that script into your repo and get your client ID using the same steps for NodeJS (this script), it should trigger the DAG just as NodeJS did.