How to run jupyter notebook in airflow

Question

My code is written in jupyter and saved as .ipynb format.

We want to use airflow to schedule the execution and define the dependencies.

How can the notebooks be executed in airflow?

I know I can convert them to python files first but the graphs generated on the fly will be difficult to handle.

Is there are any easier solution? Thanks

Answer 1

Another alternative is to use Ploomner (disclaimer: I'm the author). It uses papermill under the hood to build multi-stage pipelines. Tasks can be notebooks, scripts, functions, or any combination of them. You can run locally, Airflow, or Kubernetes (using Argo workflows).

This is how a pipeline declaration looks like:

tasks:
  - source: notebook.ipynb
    product:
      nb: output.html
      data: output.csv

  - source: another.ipynb
    product:
      nb: another.html
      data: another.csv

• Repository
• Exporting to Airflow
• Exporting to Kubernetes
• Sample pipelines