Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Unable to connect with azure blob storage with local hadoop

While trying to connect the local hadoop with the AZURE BLOB storage (ie using the blob storage as HDFS)with Hadoop Version - 2.7.1, It throws exception

Here i have successfully formed the local cluster by setting the property

<property>
    <name>fs.default.name</name>
    <value>wasb://[email protected]</value>
</property>

and followed by its key value for blob storage in core-site.xml.

while listing the file or making HDFS operations to the blob storage,getting the follwing Exception as

 ls: No FileSystem for scheme: wasb

Anyone please guide me to resolve the above issue.

like image 822
karthik Avatar asked Aug 28 '15 05:08

karthik


People also ask

How do I connect to azure storage blob?

Under the search box, select Built-in. In the search box, enter Azure blob. From the Triggers list, select the built-in trigger named When a blob is Added or Modified in Azure Storage. If you're prompted for connection details, create a connection to your Azure Storage account.

Is azure Blob Storage Hadoop?

The Azure Blob Storage interface for Hadoop supports two kinds of blobs, block blobs and page blobs.

What does the Wasb prefix mean and how does it relate to HDFS?

About. Windows Azure Storage Blob (WASB) is an file system implemented as an extension built on top of the HDFS APIs and is in many ways HDFS. The WASB variation uses: SSL certificates for improved security. the storage accounts in WASB to load data instead of from local disks in HDFS.

How do I access blob storage from URL?

By default, the URL for accessing the Blob service in a storage account is https://<your account name>. blob.core.windows.net. You can map your own domain or subdomain to the Blob service for your storage account so that users can reach it using the custom domain or subdomain.


1 Answers

  1. You need to add hadoop-azure.jar in the hadoop classpath.
  2. Download azure-storage-java project and build it using maven then you will get azure-storage-3.0.0.jar. Add the built jar in hadoop classpath.

Now you can do all the operations. Refer Azure Blob Storage documentation for more information.

like image 88
Kumar Avatar answered Oct 20 '22 10:10

Kumar