Apache AIrflow KubernetesExecutor and KubernetesPodOperator: xcom pushes not working

Question

Have got an Apache Airflow instance placed in kubernetes cluster: webserver, scheduler and postgresql. Using custom helm charts built upon bitnami's with some changes.

Airflow is working with KubernetesExecutor. All my DAGs are PythonOperator and KubernetesPodOperator (former DockerOperator - before k8s). Xcom pushes work correctly only with PythonOperator, but with KubernetesPodOperator I'm getting errors at the end of its execution (all the dags are affected):

[2019-12-06 15:12:40,116] {logging_mixin.py:112} INFO - [2019-12-06 15:12:40,116] {pod_launcher.py:217} INFO - Running command... cat /airflow/xcom/return.json
[2019-12-06 15:12:40,201] {logging_mixin.py:112} INFO - [2019-12-06 15:12:40,201] {pod_launcher.py:224} INFO - cat: can't open '/airflow/xcom/return.json': No such file or directory

So it seems that this file is not created.

I've also tried to override post_execute method to create this file there and to json.dump the results, but it didn't help - this error still persists.

Would appreciate for any suggestions on how to resolve it.

UPDATE: I've also copy-pasted this code to my DAG https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L315, and I'm still getting this error even using apache/airflow code for unit tests.

Also have to mention that my kubernetes version is 1.11.10. Minikube 1.5.2

Answer 1

Changed the database (PostgreSQL) dependency and version to the newer one and got it working.