Airflow - got an unexpected keyword argument 'conf'

Question

I'm learning Apache Airflow to implement it at my workplace, I stumbled on a problem, when trying to pass parameter to function like this (I followed the documentation)

from airflow import DAG
import pendulum
from datetime import datetime, timedelta
from airflow.operators.python_operator import PythonOperator

args = {
    "owner": "airflow",
    "start_date": pendulum.datetime(year=2020, month=10, day=5, tzinfo='Asia/Shanghai'),
    "retries": 5,
    "retry_delay": timedelta(minutes=3)
}

dag = DAG(
    "example_dag_v2",
    schedule_interval="@daily",
    default_args=args
)

def my_mult_function(number):
    return number*number

mult_task = PythonOperator(
    task_id = 'mult_task',
    provide_context=True,
    python_callable=my_mult_function,
    op_kwargs={'number': 5},
    dag = dag
)

mult_task

I keep getting this error

TypeError: my_mult_function() got an unexpected keyword argument 'conf'

where did I do wrong ?

Solution:

so i found the solution but still dont understand why the solution is

def my_mult_function(number, **kwargs):
    return number*number

i passed **kwargs on the parameters, and it works! But i still dont understand why i need to pass the **kwargs ?

Answer 1

You have set provide_context=True so PythonOperator will send the execute context to your python_callable. So a generic catch all keyword arguments, **kwargs fixes the issue.

https://github.com/apache/airflow/blob/v1-10-stable/airflow/operators/python_operator.py#L108.

If you are not going to use anything from the context then set provide_context=False.