How to add template variable in the filename of an EmailOperator task? (Airflow)

Question

I can't seem to get this to work.

I am trying to send daily a given file, whose name is like 'file_{{ds_nodash}}.csv'.

The problem is that I can't seem to add this name as the filename, since it seems it cant be used. In the text of the email or the subject works perfectly, not not on the name.

Here is the dag as an example:

local_file = 'file-{{ds_nodash}}.csv'

send_stats_csv = EmailOperator(
    task_id='send-stats-csv',
    to=['[email protected]'],
    subject='Subject - {{ ds }}',
    html_content='Here is the new file.',
    files=[local_file],
    dag=dag)

Error code: No such file or directory: u'file-{{ds_nodash}}.csv'

If i write it literally, with its given date, it works flawlessly.

Where am I wrong? How should I go about this?

Any help would be appreciated.

Thanks.

P.D. Copy paste from airflow's documentation - "The Airflow engine passes a few variables by default that are accessible in all templates". https://airflow.incubator.apache.org/code.html

If I understood correctly, these variables are accessible in execution, so if i am executing the dag, the file should be found right? I've tried both testing the task or backfilling the dag with no success.

Answer 1

Airflow Operators define what fields are template fields. For the EmailOperator only the subject and html_content fields are set as templates.

class EmailOperator(BaseOperator):
    ...
    template_fields = ('subject', 'html_content')
    template_ext = ('.html',)

See: https://airflow.incubator.apache.org/_modules/email_operator.html

From the Airflow Gotcha's Page (https://gtoonstra.github.io/etl-with-airflow/gotchas.html)

Not all parameters in operators are templated, so you cannot use Jinja templates everywhere. The Jinja templates only work for those fields in operators where it’s listed in the template_fields...

To get this to work, you would have to derive a new class from EmailOperator and add in templating for the files array.