Create Azure Databricks Token using ARM template

Question

I need to create a token in Azure Databricks using ARM template. I am able to create Azure Databricks using ARM template but unable to create token in Azure Databricks using ARM template

Following is the template which i have used to create Azure Databricks

{
"$schema": "https://schema.management.azure.com/schemas/2015-01- 
01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"type": "string",
  "metadata": {
    "description": "The name of the Azure Databricks workspace to create."
  }
},
"pricingTier": {
  "type": "string",
  "defaultValue": "premium",
  "allowedValues": [
    "standard",
    "premium"
  ],
  "metadata": {
    "description": "The pricing tier of workspace."
  }
},
"location": {
  "type": "string",
  "defaultValue": "[resourceGroup().location]",
  "metadata": {
    "description": "Location for all resources."
  }
}
},
"variables": {
"managedResourceGroupName": "[concat('databricks-rg-', 
parameters('workspaceName'), '-', uniqueString(parameters('workspaceName'), 
resourceGroup().id))]"
},
"resources": [
{
  "type": "Microsoft.Databricks/workspaces",
  "name": "[parameters('workspaceName')]",
  "location": "[parameters('location')]",
  "apiVersion": "2018-04-01",
  "sku": {
    "name": "[parameters('pricingTier')]"
  },
  "properties": {
    "ManagedResourceGroupId": "[concat(subscription().id, '/resourceGroups/', variables('managedResourceGroupName'))]"
  }
}
],
"outputs": {
"workspace": {
  "type": "object",
  "value": "[reference(resourceId('Microsoft.Databricks/workspaces', parameters('workspaceName')))]"
}
}
}

Kindly let me know how to create tokens in Azure Databricks using ARM template

Answer 1

I see in a comment that you ask if it is possible to create a token using a script. It is now possible!

Databricks has a token API: https://docs.databricks.com/dev-tools/api/latest/tokens.html

Check out this blog: https://cloudarchitected.com/2020/01/using-azure-ad-with-the-azure-databricks-api/

It shows how easy it is to create a databricks token using AAD, and a few other methods.

I have some Python Code that I use automate this task. I would extend it to automatically add the token to a key vault of some sort. Here is a sample:

import requests
import adal
import json

# set variables 
clientId = "<Service Principal Id>"
tenantId = "<Tenant Id>"
clientSecret = "<Service Principal Secret>"
subscription_id = "<Subscription Id>"
resource_group = "<Resource Group Name>"
databricks_workspace = "<Databricks Workspace Name>"
dbricks_location = "<Databricks Azure Region i.e. westus>"



# Acquire a token to authenticate against Azure management API
authority_url = 'https://login.microsoftonline.com/'+tenantId
context = adal.AuthenticationContext(authority_url)
token = context.acquire_token_with_client_credentials(
    resource='https://management.core.windows.net/',
    client_id=clientId,
    client_secret=clientSecret
)
azToken = token.get('accessToken')



# Acquire a token to authenticate against the Azure Databricks Resource
token = context.acquire_token_with_client_credentials(
    resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
    client_id=clientId,
    client_secret=clientSecret
)
adbToken = token.get('accessToken')


# Format Request API Url
dbricks_api = "https://{}.azuredatabricks.net/api/2.0".format(dbricks_location)


# Request Authentication
dbricks_auth = {
    "Authorization": "Bearer {}".format(adbToken),
    "X-Databricks-Azure-SP-Management-Token": azToken,
    "X-Databricks-Azure-Workspace-Resource-Id": ("/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Databricks/workspaces/{}".format(subscription_id, resource_group, databricks_workspace) )
    }


# Optional Paramters 
payload = {
    "comment": "This token is generated through AAD and Databricks APIs", # optional parameter
    # "lifetime_seconds": 3600 # optional parameter. If not passed then it is indefinte
}


# Request and Send Data to Create a Databricks Token
data = requests.post("{}/token/create".format(dbricks_api), headers= dbricks_auth, json=payload)

# display the response data
data.status_code
data.content

# Decode response, get token, and print token
dict_content = json.loads(data.content.decode('utf-8'))
token = dict_content.get('token_value')
print("This is the databricks token: {}".format(token))

Answer 2

This isn't possible today. It is a requested feature here on uservoice https://feedback.azure.com/forums/909463-azure-databricks/suggestions/35257819-expose-api-key-during-arm-deployment

(Please upvote)

Currently you have to log into the web UI manually and generate a token. Even the REST API doesn't support this.

Answer 3

You can actually use azure.databricks.cicd.tools in your CD pipeline to create a new bearer token.

You need to use Connect-Databricks to connect to your workspace first. I usually use the AADwithOrgId method to authenticate to the Databricks workspace:

Connect-Databricks -Region <String> -ApplicationId <String> -Secret <String> -DatabricksOrgId <String> -TenantId <String>

The service principal should have the Contributor role on your resource group and be added a an admin in your Databricks workspace.