Azure Data Factory falling behind schedule silently

Question

I have a number of activities running on ADF with some running every day, some hourly and one every 15 minutes.

I found the way to set up alerts in ADF so that failing activities will trigger an email. I have not however found the way to create more detailed custom alerts.

In this case a task that runs every 15 minutes

"scheduler": {
                "frequency": "Minute",
                "interval": 15
}

Was set to run one at a time

"policy": {
                "concurrency": 1
            },

Unfortunately the activity became locked indefinitely for a couple days. Probably on a resource lock. This caused all the time slice to stay in pending state. Waiting on concurrency. Since the initial activity slice did not fail, I got no alert and no warning.

Does anyone have an idea how to monitor failures that aren't failures in ADF like if a slice misses schedule?

Answer 1

One way to do it is to turn your issues into failures.

You can add timeout property into pipeline execution policy:

"policy": {
    "concurrency": 1,
    "timeout":"00:15:00"
}

With this timeout your pipeline execution and related dataset slice will become failed after 15 minutes.