How to get the Azure Data Factory parameters into the ARM template parameters file (ARMTemplateParametersForFactory.json) after publishing

Question

I am trying to create my Azure DevOps release pipeline for Azure Data Factory.

I have followed the rather cryptic guide from Microsoft (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment ) regarding adding additional parameters to the ARM template that gets generated when you do a publish (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template )

Created a arm-template-parameters-definition.json file in the route of the master branch. When I do a publish, the ARMTemplateParametersForFactory.json in the adf_publish branch remains completely unchanged. I have tried many configurations.

I have defined some Pipeline Parameters in Data Factory and want them to be configurable in my deployment pipeline. Seems like an obvious requirement to me.

Have I missed something fundamental? Help please!

The JSON is as follows:

{
    "Microsoft.DataFactory/factories/pipelines": {
        "*": {
            "properties": {
                "parameters": {
                        "*": "="                
                }
            }
        }
    },
    "Microsoft.DataFactory/factories/integrationRuntimes": {
        "*": "="
    },
    "Microsoft.DataFactory/factories/triggers": {},
    "Microsoft.DataFactory/factories/linkedServices": {},
    "Microsoft.DataFactory/factories/datasets": {}
}

Answer 1

I've been struggling with this for a few days and did not found a lot of info, so here what I've found out. You have to put the arm-template-parameters-definition.json in the configured root folder of your collaboration branch:

So in my example, it has to look like this:

If you work in a separate branch, you can test your configuration by downloading the arm templates from the data factory. When you make a change in the parameters-definition you have to reload your browser screen (f5) to refresh the configuration.

If you really want to parameterize all of the parameters in all of the pipelines, the following should work:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters":{
            "*":{
                "defaultValue":"="
            }
        }
    }
}

I prefer specifying the parameters that I want to parameterize:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters":{
            "LogicApp_RemoveFileFromADLSURL":{
                "defaultValue":"=:-LogicApp_RemoveFileFromADLSURL:"
            },
            "LogicApp_RemoveBlob":{
                "defaultValue":"=:-LogicApp_RemoveBlob:"
            }
        }
    }
}

Answer 2

You've got the right idea, but the arm-template-parameters-definition.json file needs to follow the hierarchy of the element you want to parameterize.

Here is my pipeline activity I want to parameterize. The "url" should change based on the environment it's deployed in

{
    "name": "[concat(parameters('factoryName'), '/ExecuteSPForNetPriceExpiringContractsReport')]",
    "type": "Microsoft.DataFactory/factories/pipelines",
    "apiVersion": "2018-06-01",
    "properties": {
        "description": "",
        "activities": [
            {
                "name": "NetPriceExpiringContractsReport",
                "description": "Passing values to the Logic App to generate the CSV file.",
                "type": "WebActivity",
                "typeProperties": {
                    "url": "[parameters('ExecuteSPForNetPriceExpiringContractsReport_properties_1_typeProperties')]",
                    "method": "POST",
                    "headers": {
                        "Content-Type": "application/json"
                    },
                    "body": {
                        "resultSet": "@activity('NetPriceExpiringContractsReportLookup').output"
                    }
                }
            }
        ]
    }
}

Here is the arm-template-parameters-definition.json file that turns that URL into a parameter.

{
   "Microsoft.DataFactory/factories/pipelines": {
        "properties": {
            "activities": [{
                "typeProperties": {
                    "url": "-::string"
                }
            }]
        }
    },
    "Microsoft.DataFactory/factories/integrationRuntimes": {},
    "Microsoft.DataFactory/factories/triggers": {},
    "Microsoft.DataFactory/factories/linkedServices": {
        "*": "="
    },
    "Microsoft.DataFactory/factories/datasets": {
        "*": "="
    }
}

So basically in the pipelines of the ARM template, it looks for properties -> activities -> typeProperties -> url in the JSON and parameterizes it.