Azure Data lake VS Azure HDInsight

Question

I was going through the Microsoft documents:

https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview

I'm new to Azure Data lake and HDInsight. There is a statement in the URL which tells that

"Azure Data Lake Store can be accessed from Hadoop (available with HDInsight cluster) using the WebHDFS-compatible REST APIs."

As per my initial understanding, Data lake store is a store in which any kind of data can be stored. I think, HDInsight also kind of does the same thing.

My question is what is the difference between Azure Data lake and Azure HDInsight? If HDInsight can be used for file storage or any kind of storage then Why to use Data Lake?It would be great if some one could clarify this in details. Thanks.

Answer 1

In nutshell,

Hdinsight is a managed hadoop service (to provide compute support)

Azure Data lake(ADL) is a managed storage service (to provide large amount of storage support)

(Instead of ADL, you can alternatively choose to use Blobs in HDinsight, but Blobs have some limitations (like file streaming to storage via hdinsight cluster is not supported)

Here is the definition from Azure documentation (below):

Azure uses "decomposed hardware method"

You can relate or assume HDinsight as a Hadoop Cluster, Azure Data lake (ADL) as HDFS. But they are detached.

If you want to relate with AWS, HDInsight is equivalent to EMR and ADL is equivalent to EMRFS or S3

If you terminate the cluster, ADL storage stays with the files stored in it. You can access the storage directly using another service or tool (like Azure Data bricks) or you can create one another hdinsight cluster on top of the data.

Hdinsight access the ADL using adl:// , and hdinsight never store the file blocks in the nodes (like Hadoop does), rather it has mappings to storage service.