Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines?

Question

Background : I have scheduled pipelines running for copying data from source to destination. This is scheduled to run daily at a specific time.

Problem : The input dataset to the pipeline is external and not available at specific time intervals. This means the copy activity will have to wait until the Scheduled Start time mentioned in the Pipeline to kickoff. Considering the volume of data, I don't want to waste my time here.

Requirement : At any given time I have access to the time when my input data set is available. With this in hand, I want to know how to trigger a ADF Pipeline from C# though its scheduled to start only at a specific time.

Answer 1

I ran into this same issue, I needed to run my pipeline only when a local job was completed. To do that I modified the local job to kick off the pipeline as its last step. I have a write up here on how to start an ADF pipeline with C#. Here is the link to the ADF developer reference which might also be helpful. I also have an example here on how to trigger ADF pipelines from Azure Functions, if you are interested. This is using the same code from the first example but I get the benefit of running the whole process in the cloud and the ability to use the azure function scheduler.

Here is the relevant method to modify the pipeline. You would need to change the start and end dates based on when you want the slice to run.

public void StartPipeline(string resourceGroup, string dataFactory, string pipelineName, DateTime slice)
    {
        var pipeline = inner_client.Pipelines.Get(resourceGroup, dataFactory, pipelineName);

        pipeline.Pipeline.Properties.Start = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T00:00:00Z");
        pipeline.Pipeline.Properties.End = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T23:59:59Z");
        pipeline.Pipeline.Properties.IsPaused = false;

        inner_client.Pipelines.CreateOrUpdate(resourceGroup, dataFactory, new PipelineCreateOrUpdateParameters()
        {
            Pipeline = pipeline.Pipeline
        });
    }

Answer 2

To trigger ADF you need to have input dataset in 'Ready' state. If it is in ready state you can manually go to Monitoring tab to manually 'Re-Run', if input dataset is not ready then you need to make that dataset ready to manually start ADF.