org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG exception with phoenix 4.5.2

Question

I've been trying to integrate Phoenix 4.5.2 to my existing hadoop cluster.

Hadoop Version : 2.7.1
HBase Version : 1.1.2

When I try to create table from my phoenix client I'm getting following exception. But I'm able to create table successfully from HBase console.

org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG
    at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1051)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1014)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1259)
    at org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(DelegateConnectionQueryServices.java:113)
    at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1937)
    at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:751)
    at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:186)
    at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:320)
    at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:312)
    at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
    at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:310)
    at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1422)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1927)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1896)
    at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1896)
    at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:180)
    at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:132)
    at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:151)
    at org.apache.tools.ant.taskdefs.JDBCTask.getConnection(JDBCTask.java:370)
    at org.apache.tools.ant.taskdefs.SQLExec.getConnection(SQLExec.java:940)
    at org.apache.tools.ant.taskdefs.SQLExec.execute(SQLExec.java:612)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform(Task.java:348)
    at org.apache.tools.ant.Target.execute(Target.java:435)
    at org.apache.tools.ant.Target.performTasks(Target.java:456)
    at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
    at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
    at org.apache.tools.ant.Project.executeTargets(Project.java:1248)
    at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:440)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform(Task.java:348)
    at org.apache.tools.ant.Target.execute(Target.java:435)
    at org.apache.tools.ant.Target.performTasks(Target.java:456)
    at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
    at org.apache.tools.ant.Project.executeTarget(Project.java:1364)
    at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
    at org.apache.tools.ant.Project.executeTargets(Project.java:1248)
    at org.apache.tools.ant.Main.runBuild(Main.java:851)
    at org.apache.tools.ant.Main.startAnt(Main.java:235)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Caused by: org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1139)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1096)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:931)
    at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
    at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:496)
    at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:736)
    at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:706)
    at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1760)
    at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1715)
    at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1695)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1034)
    ... 49 more

Please suggest what's going wrong here and Phoenix 4.5.2 is compatible with HBase 1.1.2 or not.

Answer 1

Clearing zookeeper will solve the problem.

bin/hbase clean --cleanZk

Note:- you need to shutdown your master and regionserver before using above command.

Answer 2

Here's what you need to do:

1. Login into hbase zookeeper cli: hbase zkcli
2. check to see if the SYSTEM.CATALOG, SYSTEM.SEQUENCE, SYSTEM.STATS, and SYSTEM.FUNCTION exists by issuing: ls /hbase/table
3. If so... then this is the solution:)
4. Remove all the aforementioned directories: i.e. from the same cli issue rmr /hbase/table/SYSTEM.CATALOG. Do this for all four of the aforementioned files
5. Retry saline... and smile

Answer 3

I have an embedded HBase, I had this error after deleting manually HBase data, so I have had to clean Zookeeper and the rest of HBase with:

bin/hbase clean --cleanAll

That solved the problem for me.