How to solve grpc Deadline Exceeded error？

Question

I have grpc server which write by go and a python client, sometimes, the errors occurred as below:

eggs/grpcio-1.0.0-py2.7-linux-x86_64.egg/grpc/_channel.py\", line 432, in _end_unary_response_blocking\n    raise _Rendezvous(state, None, None, deadline)\nInternalServerError: Deadline Exceeded\n"}

grpc Deadlines concept：

gRPC allows clients to specify a deadline value when calling a remote method. This specifies how long the client wants to wait for a response from the server before the RPC finishes with the error DEADLINE_EXCEEDED. On the server side, the server can query the deadline to see if a particular method has timed out, or how much time is left to complete the method.

How the deadline is specified varies from language to language - for example, a deadline value is always required in Python, and not all languages have a default deadline.

is there way to solve this error?

Answer 1

As mentioned in the comment above, deadlines can be just about anything between the client and server, including the network and implementation of the server. As you are talking over a network, deadlines should be expected occasionally, for example during package losses. I general, there are a couple of options on what you can do here:

• Optimize the server implementation to handle your requests faster.
• In case the client sets a short deadline, increase this.
• Handle the deadline errors gracefully by:
  • Retrying. Just make sure to use an exponential back-off, so you don't make the problem worse in case of a server overload situation.
  • Report an error up-streams.
  • Sometimes it may also be possible to fall back to not use that grpc call and degrade the experience.