Protobuf Error : Hbase createTable & put in java code(protobuf-LiteralByteString)

Question

I am developing java application with kafka, spark stream & hbase. After I compile code by maven clean install, when I run my application I met the following error:

• hadoop ver : 2.7.3
• HBase ver: hbase-0.98.24-hadoop2
• Spark : 2.1.0

Although I've already checked & applied the previous answers in this site, I still met the problem.

Thank you ...

`Exception in thread "main" org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: com/google/protobuf/LiteralByteString
at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:225)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:133)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:190)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:185)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:111)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:789)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:542)
at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:310)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:317)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:331)

`

Caused by: java.lang.NoClassDefFoundError: com/google/protobuf/LiteralByteString
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

Answer 1

Apache Hadoop 2.7.3 has a dependency on Protobuf 2.5.0. Reviewing the source code for Protobuf 2.5.0, the class com.google.protobuf.LiteralByteString is visible here:

https://github.com/google/protobuf/blob/v2.5.0/java/src/main/java/com/google/protobuf/LiteralByteString.java

However, this class is not present in later versions of Protobuf. For example, here is the com/google/protobuf directory in the recent 3.1.0 version:

https://github.com/google/protobuf/tree/v3.1.0/java/core/src/main/java/com/google/protobuf

Based on this, it is likely that your application has picked up a version of Protobuf other than 2.5.0 and put it on the classpath. I recommend running mvn dependency:tree on your application to see if it's picking up an incompatible Protobuf dependency (possibly transitively). If it looks fine in your application's Maven dependencies, then perhaps something is overriding the classpath at runtime when you launch the application and picking up an incorrect version of Protobuf.