how to transfer data between Azure tables and Hadoop on Azure

Question

I would like to import data from Azure table and then export to Hadoop cluster on Azure. Currently I am downloading the data from Azure table into .csv and then uploading the .csv to Hadoop cluster. I want to know whether there is any way to do it directly.

Thanks, Kiran

Answer 1

I have written a HiveStorageHandler which will let you create Hive Table which are backed by Azure Storage Tables. It's available on github:

https://github.com/simonellistonball/hive-azuretables

Note that you will also need the Microsoft Azure SDK for Java in your hadoop classpath for this to work. The easiest way to do that is probably to install it somewhere central like /usr/lib/hadoop/lib/.

If you do a mvn package on my repo, you should get something you can then load with a hive ADD JAR. The Readme for the repo contains a sample query showing how to create a Hive table that connects to Azure.