Data Governance solution for Databricks, Synapse and ADLS gen2

Question

I'm new to data governance, forgive me if question lack some information.

Objective

We're building data lake & enterprise data warehouse from scratch for mid-size telecom company on Azure platform. We're using ADLS gen2, Databricks and Synapse for our ETL processing, data science, ML & QA activities.

We already have about a hunder of input tables and 25 TB/yearly. In future we're expecting more.

Business has a strong requirements incline towards cloud-agnostic solutions. Still they are okay with Databricks since it's available on AWS and Azure.

Question

What is the best Data Governance solution for our stack and requirements?

My workarrounds

I haven't used any data governance solutions yet. I like AWS Data Lake solution, since it provide basic functionality out-of-the-box. AFAIK, Azure Data Catalog is outdated, because it doesn't support ADLS gen2.

After very quick googling I found three options:

1. Databricks Privacera
2. Databricks Immuta
3. Apache Ranger & Apache Atlas.

Currently I'm not even sure if the 3rd option has full support for our Azure stack. Moreover, it will have much bigger development (infrastructure definition) effort. So is there any reasons I should look into Ranger/Atlas direction?

What are the reasons to prefer Privacera over Immuta and vice versa?

Are there any other options I should evaluate?

What is already done

From Data Governance perspective we have done only the following things:

1. Define data zones inside ADLS
2. Apply encryption/obfuscation for sensitive data (due to GDPR requirements).
3. Implemented Row-Level Security (RLS) at Synapse and Power BI layers
4. Custom audit framework for logging what & when was persisted

Things to be done

1. Data lineage and single source of truth. Even at 4 months from the start, it become a pain-point to understand dependencies between data sets. The lineage information is stored inside Confluence, it's hard to maintain and continuously update in multiple places. Even now it's outdated in some places.
2. Security. Business users may do some data exploration in Databricks Notebooks in future. We need RLS for Databricks.
3. Data Life Cycle management.
4. Maybe other data governance related stuff, such as data quality, etc.

Answer 1

I am currently exploring Immuta and Privacera, so I can't yet comment in detail on differences between these two. So far, Immuta gave me better impression with it's elegant policy based setup.

Still, there are ways to solve some of the issues you mentioned above without buying an external component:

1. Security

• For RLS, consider using Table ACLs, and giving access only to certain Hive views.

• For getting access to data inside ADLS, look at enabling password pass-through on clusters. Unfortunately, then you disable Scala.

• You still need to setup permissions on Azure Data Lake Gen 2, which is awful experience for giving permissions on existing child items.

• Please avoid creating dataset copies with columns/rows subsets, as data duplication is never a good idea.

2. Lineage

• One option would be to look into Apache Atlas & Spline. Here is one example how to set this up https://medium.com/@reenugrewal/data-lineage-tracking-using-spline-on-atlas-via-event-hub-6816be0fd5c7
• Unfortunately, Spline is still under development, even reproducing the setup mention in the article is not straight forward. Good news that Apache Atlas 3.0 has many available definitions to Azure Data Lake Gen 2 and other sources
• In a few projects, I ended up creating custom logging of reads/writes (seems like you went on this path also). Based on these logs, I created a Power BI report to visualize the lineage.
• Consider using Azure Data Factory for orchestration. With a proper ADF pipeline structure, you can have a high level lineage and help you see dependencies and rerun failed activities. You can read a bit more here: https://mrpaulandrew.com/2020/07/01/adf-procfwk-v1-8-complete-pipeline-dependency-chains-for-failure-handling/
• Take a look at Marquez https://marquezproject.github.io/marquez/. Small open-source library that has some nice features, including data lineage.

3. Data quality

• Investigate Amazon Deequ - Scala only so far but has some nice predefined data quality functions.
• In many projects, we ended up with writing integration tests, checking data quality between moving from bronze (raw) to silver (standardized). Nothing fancy, pure PySpark.

4. Data life cycle management

• One option is to use native data lake storage lifecycle management. That's not a viable alternative behind Delta/Parquet formats.

• If you use Delta format, you can easier apply retention or pseudoanonymize

• Second option, imagine that you have a table with information about all datasets (dataset_friendly_name, path, retention time, zone, sensitive_columns, owner, etc.). Your Databricks users use a small wrapper to read/write:

  DataWrapper.Read("dataset_friendly_name")

  DataWrapper.Write("destination_dataset_friendly_name")

It's up to you then to implement the logging, data loading behind the scenes. In addition you can skip sensitive_columns, acts based on retention time (both available in dataset info table). Requires quite some effort

• You can always expand this table to more advanced schema, add extra information about pipelines, dependencies, etc. (see 2.4)

Hopefully you find something useful in my answer. It would be interesting to know which path you took.

Answer 2

To better understand option #2 that you cited for data governance on Azure, here is a how-to tutorial demonstrating the experience of applying RLS on Databricks; a related Databricks video demo; and other data governance tutorials.

Full disclosure: My team produces content for data engineers at Immuta and I hope this helps save you some time in your research.