Azure Databricks CI/CD

Question

I am referring this link to implement CI/CD in my Databricks environment. I have set up GitHub and synced my notebook with my branch. I have created a staging and production folder for each environment. Currently I manually Import and export my notebook from master to staging folder and ran a job. But I have some questions -

1] How to automate build after each merge to central repository?

2] How to push latest changes to multiple environment?

Answer 1

Take a look at this PowerShell module to import/export notebooks https://github.com/DataThirstLtd/azure.databricks.cicd.tools/.

The basic flow is:

• Dev your notebooks as normal online
• Use Export-DatabricksFolder the notebooks to your local disk and commit to repo
• CI build runs and deploys using the Import-DatabricksFolder command

As you have used the GitHub integration you can skip the Export.