Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Moving - not copying - data in Azure Data Factory

I'd like to set up an Azure Data Factory pipeline which performs a move (i.e. copy, verify, delete) operation rather than just a copy operation between Blob Storage and a Data Lake Store. I cannot seem to find any detail on how to do this.

like image 297
Sam Avatar asked Jan 19 '17 06:01

Sam


People also ask

How do you solve performance problems in ADF?

The solution to this problem can be found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session. With a low memory consumption, a more responsive, stable and scalable ADF application can be delivered.


1 Answers

Azure Data Factory does not have a built-in activity or option to Move files as opposed to Copy them. You can however do this with a Custom Activity.

This example on github shows how to do this with Azure Blob:

...
blob.DeleteIfExists();
...

https://github.com/Azure/Azure-DataFactory/tree/master/Samples/DeleteBlobFileFolderCustomActivity

If you feel this is an important feature, please add a feedback request:

https://feedback.azure.com/forums/270578-data-factory

A Delete activity has been added recently:

https://azure.microsoft.com/en-us/blog/clean-up-files-by-built-in-delete-activity-in-azure-data-factory/

like image 185
wBob Avatar answered Oct 13 '22 00:10

wBob