TemporaryBackendException while using JanusGraph Java API and HBase

Question

I am using JanusGraph Java API, with HBase as storage backend

One thing I have to mention is that I can successfully run the same commands in JanusGraph shell, gremlin.sh

The log before the error is below, which happens during the opening of the graph instance, JanusGraphFactory.open(jg.properties)

07:20:46.169 [main-SendThread(cdh-master1:2181)] DEBUG org.apache.zookeeper.ClientCnxn - 
Reading reply sessionid:0x263f89ba9d56ce2, 
packet:: clientPath:null serverPath:null finished:false header:: 3,8  replyHeader:: 3,55834924519,0  
request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,
'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'master,'running,'recovering-regions,'draining,'namespace,'hbaseid,'table} 
07:20:46.182 [main] WARN  o.j.d.hbase.HBaseStoreManager - Unexpected exception during getDeployment()

and the error is

java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1377) [jar:rsrc:janusgraph-core-0.2.0.jar!/:na]
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164) [jar:rsrc:janusgraph-core-0.2.0.jar!/:na]
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133) [jar:rsrc:janusgraph-core-0.2.0.jar!/:na]
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80) [jar:rsrc:janusgraph-core-0.2.0.jar!/:na]

I have no idea with this TemporaryBackendException, has anyone met this before?

part of the janusgraph config is

gremlin.graph=org.janusgraph.core.JanusGraphFactory
cluster.max-partitions=64
storage.backend=hbase
storage.batch-loading=true
storage.hostname=cdh-master1,cdh-master2,cdh-master3

where cdh-master1,2,3 are zookeeper quorum, version JanusGraph 2.0, HBase 1.2

The full stack trace is

at my.graph.Graph.<clinit>(Graph.java:7) [rsrc:./:na]
at my.graph.App.main(App.java:24) [rsrc:./:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58) [janus.jar:na]
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
... 12 common frames omitted
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:303) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716) ~[jar:rsrc:janusgraph-hbase-0.2.0.jar!/:na]
... 14 common frames omitted
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[jar:rsrc:hbase-client-1.1.2.jar!/:na]
... 24 common frames omitted

part of my pom.mxl including the dependency information is

<dependency>
    <groupId>org.janusgraph</groupId>
    <artifactId>janusgraph-core</artifactId>
    <version>0.2.0</version>
</dependency>
<dependency>
    <groupId>org.janusgraph</groupId>
    <artifactId>janusgraph-hbase</artifactId>
    <version>0.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-client</artifactId>
    <version>1.1.2</version>
</dependency>

Answer 1

I posted an answer here previously:

This is a Guava version conflict issue. The JanusGraph HBase example does not run into a Guava version error because of its specific dependencies listed and the order of those dependencies.

    <dependency>
        <groupId>org.janusgraph</groupId>
        <artifactId>janusgraph-hbase</artifactId>
        <version>${project.version}</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-shaded-client</artifactId>
        <version>${hbase100.version}</version>
        <scope>runtime</scope>
    </dependency>

Note the usage of hbase-shaded-client (fixed with Issue #512) rather than hbase-client in your pom.xml which has a shaded version of Guava for use with HBase. Shading renames the package names of some of the dependencies, and this allows you to have different versions of Guava on the classpath without causing conflicts.

The Gremlin Console uses the hbase-shaded-client also and you can find that jar in the lib directory of the distribution.