Menu
I have a pipeline built on Azure data Factory. It has:
a "LookUp" task that has an SQL query that returns a column [CountRecs]. This columns holds a value 0 or more.
an "if" task to check this returned value. I want to fail the pipeline when the value of [CountRecs]>0
Is this possible?
983
Create a Fail activity with UISearch for Fail in the pipeline Activities pane, and drag a Fail activity to the pipeline canvas. Select the new Fail activity on the canvas if it is not already selected, and its Settings tab, to edit its details. Enter a failure message and error code.
Trigger the pipeline manuallySelect Trigger on the toolbar, and then select Trigger Now. On the Pipeline Run page, select OK. Go to the Monitor tab on the left. You see a pipeline run that is triggered by a manual trigger.
To add parameters to your data flow, click on the blank portion of the data flow canvas to see the general properties. In the settings pane, you will see a tab called Parameter. Select New to generate a new parameter. For each parameter, you must assign a name, select a type, and optionally set a default value.
Similarly, you might use a Hive activity, which runs a Hive query on an Azure HDInsight cluster, to transform or analyze your data. Data Factory supports three types of activities: data movement activities, data transformation activities, and control activities.
You could probably achieve this by having a Web Activity when your IF Condition is true ([CountRecs]>0) in which the web activity should call the below REST API to cancel the pipeline run by using the pipelinerunID (you can get this value by using dynamic expression - @pipeline().RunId)
Sample Dynamic Expression for Condition: @greater(activity('LookupTableRecordCount').output.firstRow.COUNTRECS, 0)
REST API to Cancel the Pipeline Run: POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/cancel?api-version=2018-06-01
MS Doc related to Rest API: ADF Pipeline Runs - Cancel
One other possible way is to have an invalid URL in your web activity which will fail the Web activity in-turn it will fail the IfCondition activity, which inturn will result in your pipeline to fail.
There is an existing feature request related to the same requirement in ADF user voice forum suggested by other ADF users. I would recommend you please up-vote and/or comment on this feedback which will help to increase the priority of the feature request implementation.
ADF User voice feedback related to this requirement: https://feedback.azure.com/forums/270578-data-factory/suggestions/38143873-a-new-activity-for-cancelling-the-pipeline-executi
Hope this helps.
125
As a sort-of hack-solution you can create a "Set variable" activity which incurs division by zero if a certain condition is met. I don't like it but it works.
@string(
div(
1
, if(
greater( int(variables('date_diff')), 100 )
, 0
, 1
)
)
)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
