Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure Data Factory get data from foreach value

I've a lookup activity with gets value from a container within a blob. I've a foreach activity which is connected to the lookup activity

enter image description here

and under settings it has the value:

@activity('LookupMessageType').output.value

I've another copy activity which runs within this foreach activity. It copies data from cosmos DB to Azure Data Lake.
enter image description here

This is the query in the source dataset:

select c.Tag, data.assetTag, data.timestamp, data.jsonVersion, data.messageType, data.speed from c join data in c.data
where (data.speed>  item().speed_Value) AND
(data.timestamp >= '@{formatDateTime(addhours(pipeline().TriggerTime, -1), 'yyyy-MM-ddTHH:mm:ssZ' )}' 
AND data.timestamp < '@{formatDateTime(pipeline().TriggerTime, 'yyyy-MM-ddTHH:mm:ssZ' )}')

I get an error as I run this pipeline:

{
    "errorCode": "2200",
    "message": "Failure happened on 'Source' side. ErrorCode=UserErrorDocumentDBReadError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=DocumentDb operation failed: Message: {\"errors\":[{\"severity\":\"Error\",\"location\":{\"start\":231,\"end\":235},\"code\":\"SC2005\",\"message\":\"'item' is not a recognized built-in function name.\"}]}\r\nActivityId: *redacted*, documentdb-dotnet-sdk/1.21.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0.,Source=Microsoft.DataTransfer.ClientLibrary.DocumentDb,''Type=Microsoft.Azure.Documents.BadRequestException,Message=Message: {\"errors\":[{\"severity\":\"Error\",\"location\":{\"start\":231,\"end\":235},\"code\":\"SC2005\",\"message\":\"'item' is not a recognized built-in function name.\"}]}\r\nActivityId: redacted, documentdb-dotnet-sdk/1.21.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,''Type=System.Runtime.InteropServices.COMException,Message=Exception from HRESULT: 0x800A0B00,Source=,'",
    "failureType": "UserError",
    "target": "Copy Data1"
}

It is saying item is not a built in function name. I want to use value from the foreach activity to query the source.

Furthermore, here is my dynamic filepath content

@concat('test_', item().speed, '.csv')

I get the desired dynamic file structure with this expression, if I am using static values in the query like: data.speed> 500

like image 919
MAK Avatar asked Sep 13 '18 18:09

MAK


1 Answers

So, I figured out what was wrong with the query: instead of

where (data.speed> item().speed_Value)

I had to use

where (data.speed> @{item().speed_value})
like image 67
MAK Avatar answered Sep 27 '22 18:09

MAK