Hbase Client RPC Timeout

Question

I'm running Hbase 1.0.1/Hadoop 2.5.2. I'm trying to run a scan on a table but I'm getting RPC timeouts.

I've changed the Hbase RPC timeout to 2 minutes which I can confirm frm the UI...

<property>
  <name>hbase.rpc.timeout</name>
  <value>120000</value>
  <source>hbase-site.xml</source>
</property>

... but my client is still timing out after 60s...

Caused by: java.io.IOException: Call to xxxxxxx/172.16.5.13:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2968, waitTime=60001, operationTimeout=60000 expired.
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1235)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31751)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
    ... 6 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2968, waitTime=60001, operationTimeout=60000 expired.
    at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1177)
    ... 12 more

I've tried changing the cache block size but this doesn't seem to make any difference.

Is there some other timeout that I'm missing. There are a lot of rows in the table (millions) although the scan only returns 10's of thousands but the problem only seems to be with a specific set of regions.

Answer 1

When the server receives a scan RPC request, a time limit is calculated to be half of the smaller of two values: hbase.client.scanner.timeout.period and hbase.rpc.timeout (which both default to 60000 milliseconds, or one minute). This is why after setting 2 min your scan is getting timed out in 60 seconds.

When the time limit is reached, the server returns the results it has accumulated up to that point. This result set may be empty. If your usage pattern includes that scans will take longer than a minute, you can increase these values.

To make sure the timeout period is not too short, you can configure hbase.cells.scanned.per.heartbeat.check to a minimum number of cells that must be scanned before a timeout check occurs. The default value is 10000. A smaller value causes timeout checks to occur more often.

Below links might be helpful to configure timeouts for Hbase scan:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/best-practices-timeouts-phoenix.html

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/admin_hbase_scanner_heartbeat.html#concept_xsl_dz1_jt

Answer 2

Based on the first answers. I configured the properties hbase.client.scanner.timeout.period and hbase.rpc.timeout before to open connection with the HBase Cluster:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;

Configuration conf = HBaseConfiguration.create();

conf.set("hbase.rpc.timeout", "1800000");
conf.set("hbase.client.scanner.timeout.period", "1800000");

Answer 3

Try this when you create connection in client side instead of in hbase-site.xml:

conf.set("hbase.rpc.timeout", "1800000")