I'd like to set up an Azure Data Factory pipeline which performs a move (i.e. copy, verify, delete) operation rather than just a copy operation between Blob Storage and a Data Lake Store. I cannot seem to find any detail on how to do this.
The solution to this problem can be found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session. With a low memory consumption, a more responsive, stable and scalable ADF application can be delivered.
Azure Data Factory does not have a built-in activity or option to Move files as opposed to Copy them. You can however do this with a Custom Activity.
This example on github shows how to do this with Azure Blob:
...
blob.DeleteIfExists();
...
https://github.com/Azure/Azure-DataFactory/tree/master/Samples/DeleteBlobFileFolderCustomActivity
If you feel this is an important feature, please add a feedback request:
https://feedback.azure.com/forums/270578-data-factory
A Delete activity has been added recently:
https://azure.microsoft.com/en-us/blog/clean-up-files-by-built-in-delete-activity-in-azure-data-factory/
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With