I would like to set an SLA in a Sensor Operator. The documentation is not too clear about the use of it. So I did a test using the S3KeySensor
operator which is looking for a file that does not exist. I set the SLA to 30 seconds, I was hoping to see the record after 30 seconds in the UI - in SLA misses - but it did not happen. What am I doing wrong?
inputsensor = S3KeySensor(
task_id="check_for_files_in_s3",
bucket_key="adp/backload/20136585/",
wildcard_match=True,
bucket_name="weblogs-raw",
s3_conn_id="AWS_S3_CENTRAL",
timeout=120,
poke_interval=10,
sla=timedelta(seconds=30),
dag=dag,
)
inputsensor.set_downstream(next_step)
In hours
"sla": timedelta(hours=2)
In minutes
"sla": timedelta(minutes=120)
According to the documentation SLA represents the timedelta after the schedule period is over. So if your schedule interval is '@daily'
and sla=timedelta(hours=1)
then Airflow will check for SLA misses at 1:00 AM which is when the schedule period is over plus one hour.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With