How to use hbase with Spring Boot using Java instead of XML?

Question

I have Spring Boot Hadoop and want to take advantage of the Spring HbaseTemplate. My issue is the documentation has only information about the "xml" way of the configuration and setup.

How and where do I define my configuration to hbase configuration in java as opposed to the xml as show in the official docs?

http://docs.spring.io/spring-hadoop/docs/1.0.1.RC1/reference/html/hbase.html

Answer 1

Well it is not really an expected answer but I want to develop it too much for a comment.

I carefully read the Spring for Apache Hadoop - Reference Documentation in its last released version, and if it does contain examples and details for namespace configuration, I could not find a single line on Java configuration.

My understanding on it is that only namespace configuration is currently supported for Spring for Apache Hadoop. It should certainly be possible to look at the classes supporting the namespace and hack a working configuration to find how to obtain same result with java config, but honestly I think the cost/gain ratio does not justify it. And as it is currently undocumented, you will never be sure that you did not forget something that will break later.

As Spring offers to include xml config files in a Java config Spring application, I would strongly advice you to keep all your present Java config, write the Apache Hadoop part in xml using the supplied namespace, and simply add an @ImportResource annotation to your configuration class. Say the spring hadoop configuration is hadoopContext.xml at the root of classpath, you could write :

@Configuration
...
@ImportResource("classpath:/hadoopContext.xml")
public classConfig {
...
}

Alternatively you can use a @Configuration wrapper around the xml config that you manage to be scanned by Spring :

@Configuration
@ImportResource("classpath:/hadoopContext.xml")
public class HadoopConfig {

}

Answer 2

Even though a Java-based configuration for HBase is yet not formally available (as recently as Spring Hadoop 2.2.1), there is a clean workaround. Include the relevant ZooKeeper details as properties within a Java-based Spring Hadoop configuration. Then, the hbase-configuration XML tag need not include any configuration details. By convention, it will depend on the hadoopConfiguration bean defined by the Java-based HadoopConfigConfigurer.

@Configuration
@EnableHadoop
public class MapReduceConfiguration extends SpringHadoopConfigurerAdapter {
    @Override
    public void configure(HadoopConfigConfigurer config) throws Exception {
        config
                .fileSystemUri(myConfigManager.getHadoopFsUri())
                .resourceManagerAddress(myConfigManager.getHadoopResourceManagerAddress())
                .withProperties()
                .property("hbase.zookeeper.quorum", myConfigManager.getZookeeperQuorum())
                .property("hbase.zookeeper.property.clientPort", myConfigManager.getZookeeperPort())
                .property("hbase.client.scanner.caching", "1");
    }
}

The remaining XML configuration is agnostic to the deployment environment:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:hdp="http://www.springframework.org/schema/hadoop"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
    <hdp:hbase-configuration />
</beans>

Answer 3

I wrote a simple demo project for using hbase in spring boot application without xml.

This demo mainly depends spring-data-hadoop and hbase-client. gradle dependencies:

compile('org.springframework.boot:spring-boot-starter-data-rest')
compile('org.springframework.boot:spring-boot-starter-web')
compile 'org.springframework.data:spring-data-hadoop:2.5.0.RELEASE'
compile('org.apache.hbase:hbase-client:1.3.1'){
    exclude group :'log4j',module:'log4j'
    exclude group :'org.slf4j',module:'slf4j-log4j12'
    exclude group: 'javax.servlet', module: 'servlet-api'
}
compile('org.springframework.boot:spring-boot-configuration-processor')
providedRuntime('org.springframework.boot:spring-boot-starter-tomcat')

Configure the hbase connection parameters in spring boot's application.properties (No XML!):

spring.data.hbase.zkQuorum=192.168.0.109:2181
spring.data.hbase.zkBasePath=/hbase
spring.data.hbase.rootDir=file:///home/hbase-1.2.2

class HbaseProperties.java:

@ConfigurationProperties(prefix = "spring.data.hbase")
public class HbaseProperties {
    // Addresses of all registered ZK servers.
    private String zkQuorum;

    // Location of HBase home directory
    private String rootDir;

    // Root node of this cluster in ZK.
    private String zkBasePath;

    // getters and setters...

}

HbaseConfig.java, inject the configurations into the HbaseTemplate:

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.hadoop.hbase.HbaseTemplate;

@Configuration
@EnableConfigurationProperties(HbaseProperties.class)
public class HbaseConfig {

    @Autowired
    private HbaseProperties hbaseProperties;

    @Bean
    public HbaseTemplate hbaseTemplate() {
        org.apache.hadoop.conf.Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", this.hbaseProperties.getZkQuorum());
        configuration.set("hbase.rootdir", this.hbaseProperties.getRootDir());
        configuration.set("zookeeper.znode.parent", this.hbaseProperties.getZkBasePath());
        return new HbaseTemplate(configuration);
    }

}

Service class, we can use the configured HbaseTemplate now:

import javax.annotation.PostConstruct;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.hadoop.hbase.HbaseTemplate;
import org.springframework.stereotype.Service;

import com.zql.hbasedemo.vo.Quote;

@Service
public class FeedService {
    @Autowired
    private HbaseTemplate hbaseTemplate;

    @PostConstruct
    public void test(){
        Quote quote = new Quote();
        quote.setEventType("ft");
        quote.setHandicap("4");
        quote.setMarket("OU");
        quote.setMatchId("27350208");
        quote.setSelection("OVER");
        quote.setPrice("1.93");
        saveQuote(quote);
    }

    public void saveQuote(Quote quote) {
        hbaseTemplate.put("quotes", quote.getMatchId(), "data", quote.getMarket() + ":" + quote.getSelection(),
            quote.getPrice().getBytes());
    }
}

Enjoy! : )

Answer 4

you can check out this samples
https://github.com/trisberg/bostonhadoop
http://santoshkrsh.blogspot.in/2014/01/spring-restful-service-crud-operation.html
https://github.com/spring-projects/spring-hadoop-samples/tree/master/hbase