Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure Data Factory check rowcount of copied records

I am designing a ADF pipeline that copies rows from a SQL table to a folder in Azure Data Lake. After that the rows in SQL should be deleted. But for this delete action takes place I want to know if the number rows that are copied are the same as the number of rows that were I selected in the beginning of the pipeline. Is there a way to get the rowcount fo the copy action and use this in another action (like a lookup)

Edit follow up question: Bo Xiao's answer is OK. BUt then I have a follow up question. After the copy-activity I put an If Condition with the following expression:

@activity('LookUpActivity').output.firstRow.RecordsRead ==      @{activity('copyActivity').output.rowsCopied

But then I get the error: @activity('LookUpActivity').output.firstRow.RecordsRead == @{activity('copyActivity').output.rowsCopied

Isn't it possible to compare output parameters of two activities to see if this is True?

extra edit: I just found an error in this piece of code. I forgot a "{" at the begin of the code. But then the code is still wrong. To compare two outputs from earlier activities the code must be:

@equals(activity('LookUpActivity').output.firstRow.RecordsRead,activity('copyActivity').output.rowsCopied)
like image 611
jbazelmans Avatar asked Jul 27 '18 07:07

jbazelmans


People also ask

How do you count record in ADF?

In this blog, we will learn how to get distinct rows and rows count from the data source via ADF's Mapping Data flows step by step. Step 1: Create an Azure Data Pipeline. Step 2: Add a data flow activity and name as “DistinctRows”. Step 3: Go to settings and add a new data flow.

Can Azure Data Factory have multiple pipelines?

A data factory can have one or more pipelines. A pipeline is a logical grouping of activities that together perform a task. The activities in a pipeline define actions to perform on your data. For example, you may use a copy activity to copy data from a SQL Server database to an Azure Blob Storage.

Can Azure Data Factory call stored procedure?

Invoking a stored procedure while copying data into an Azure Synapse Analytics by using a copy activity is not supported. But, you can use the stored procedure activity to invoke a stored procedure in Azure Synapse Analytics.


1 Answers

You can find copied rows in activity output as pictured below.

And you can use the output value like this:

@activity('copyActivity').output.rowsCopied

enter image description here

like image 50
Bo Xiao Avatar answered Sep 28 '22 11:09

Bo Xiao