Add custom volumeMount into Airflow worker pods (with k8s Executor)

Question

There is an option of mounting volumeMount for dags and logs under kubernetes section of airflow.conf e.g.

[kubernetes]
airflow_configmap = airflow_config
worker_container_repository = airflow
worker_container_tag = runner2
worker_container_image_pull_policy = IfNotPresent
delete_worker_pods = true
dags_volume_claim = airflow-dags-pvc
dags_volume_subpath = airflow/development/dags
logs_volume_claim = airflow-logs-pvc
logs_volume_subpath = airflow/development/logs
namespace = development

This works as expected. Means I could see worker pods successfully mounting both of these volumes and its relevant volumeMounts inside the container.

   "volumeMounts": [
      {
        "name": "airflow-dags",
        "readOnly": true,
        "mountPath": "/usr/local/airflow/dags",
        "subPath": "airflow/development/dags"
      },
      {
        "name": "airflow-logs",
        "mountPath": "/usr/local/airflow/logs",
        "subPath": "airflow/development/logs"
      },

BUT, My worker pods have dependency of picking up custom airflow plugins from directory airflow/development/plugins and airflow/development/libs. Due to which I need to add more volumeMount into the worker pod with relevant subPaths from NFS server. How can I achieve that ? I tried searching for any relevant config value but couldn't find any.

Update: I was passing executor_config into the one of the dags sensors task as executor_config={"KubernetesExecutor": {"image": "airflow:runner2"}}. By the look at code airflow/contrib/kubernetes/worker_configuration.py , it seems like if I pass in volumes and volumeMounts as part of this configs, it should work. I will try this and update here.

Answer 1

Sorry i'd to answer my own question. Maybe will help someone. Inside the dag file, where I define the task, I simply had to add the executor_config as follows;

        IngestionStatusSensor(
            task_id=...,
            executor_config={"KubernetesExecutor": {
                                  "image": "airflow:runner2",
                                  "volume_mounts": [
                                      {
                                          "name": "airflow-dags",
                                          "mountPath": "/usr/local/airflow/libs",
                                          "subPath": "airflow/development/libs"
                                      },
                                      {
                                          "name": "airflow-dags",
                                          "mountPath": "/usr/local/airflow/plugins",
                                          "subPath": "airflow/development/plugins"
                                      }],
                                  }
                             },
            dag=dag,
            ingestion_feed=table,
            poke_interval=60 * 30,
            offset=0,
            start_date=start_date
        )

where airflow-dags volume is already defined by pod creator that claims the PVC defined in configs under kubernetes section of airflow.conf e.g. dags_volume_claim = airflow-pvc.