Advantages of using HDInsights SPARK over Azure Databricks cluster

Question

I have gone through multiple documents, but not able to get the list of advantages of using HDInsigths spark cluster compared to Azure Databricks cluster. Is there any key differentiators between these two. I need basically the list of features supported by HDInsights and not supported by Azure Databricks.

Answer 1

Let me take you through a visual journey and show some screenshots. I have attached a few screenshots for Azure Spark & Azure Databricks. Azure spark is HDInsight (Hortomwork HDP) bundle on Hadoop. Databricks looks very different when you initiate the services.

You have to choose the number of nodes and configuration and rest of the services will be configured by Azure services.

If you look at the HDInsight Spark instance, it will have the following features. Refer these 2 screenshots &&

1. Apache Ranger
2. Enterprise Security Package
3. HDInsight IO Cache
4. Secure shell (SSH) access
5. HDInsight applications
6. Custom virtual network
7. Custom Hive metastore
8. Custom Oozie metastore
9. Data Lake Storage Gen1 access

On other hands, Azure Spark is a complete abstraction and gives the following feature without any configuration. Choose the resource and input basic information, the instance will start . Your instance is up and running. Your platform is ready for use, . All your notebooks, tutorial etc are available and ready for use. You don't need to think about anything else. Your final resources look like this once databricks instance is created.

The data bricks prices are very different and you can refer pricing here Databricks links for pricing is here