Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Problem with gRPC setup. Getting an intermittent RPC unavailable error

Tags:

go

rpc

grpc

I have a grpc server and client that works as expected most of the time, but do get a "transport is closing" error occasionally:

rpc error: code = Unavailable desc = transport is closing

I'm wondering if it's a problem with my setup. The client is pretty basic

connection, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
pb.NewAppClient(connection)
defer connection.Close()

and calls are made with a timeout like

ctx, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
defer cancel()

client.MyGRPCMethod(ctx, params)

One other thing I'm doing is checking the connection to see if it's either open, idle or connecting, and reusing the connection if so. Otherwise, redialing.

Nothing special configuration is happening with the server

grpc.NewServer()

Are there any common mistakes setting up a grpc client/server that I might be making?

like image 363
harumphfrog Avatar asked Oct 25 '18 15:10

harumphfrog


People also ask

What is RPC in gRPC?

A RPC is a form of Client-Server Communication that uses a function call rather than a usual HTTP call. It uses IDL (Interface Definition Language) as a form of contract on functions to be called and on the data type. RPC Architecture. If you all haven't realized it yet, the RPC in gRPC stands for Remote Procedure Call ...

What is blocking gRPC?

gRPC supports blocking client call. What this means is that once the client makes the call to the service, the client would not proceed with rest of the code execution until it gets the response back from the server. Note that a blocking client call is possible for unary calls and server streaming calls.

What is the default port for gRPC?

To open a gRPC connection to a service so you can send gRPC messages, you need to specify the host domain, which is the URL of the Cloud Run service or the custom domain mapped to that service, along with the port 443, which is the port expected to be used by gRPC.

How can I improve my gRPC performance?

There are two possible solutions: Create a separate channel for each area of high load in the application. Use a pool of gRPC channels to distribute RPCs over multiple connections (channels must have different channel args to prevent re-use so define a use-specific channel arg such as channel number).


2 Answers

After much search, I have finally come to an acceptable and logical solution to this problem.

The root-cause is this: The underlying TCP connection is closed abruptly, but neither the gRPC Client nor Server are 'notified' of this event.

The challenge is at multiple levels:

  • Kernel's management of TCP sockets
  • Any intermediary load-balancers/reverse-proxies (by Cloud Providers or otherwise) and how they manage TCP sockets
  • Your application layer itself and it's networking requirements - whether it can reuse the same connection for future requests not

My solution turned out to be fairly simple:

server = grpc.NewServer(
    grpc.KeepaliveParams(keepalive.ServerParameters{
        MaxConnectionIdle: 5 * time.Minute,           // <--- This fixes it!
    }),
)

This ensures that the gRPC server closes the underlying TCP socket gracefully itself before any abrupt kills from the kernel or intermediary servers (AWS and Google Cloud Load Balancers both have larger timeouts than 5 minutes).

The added bonus you will find here is also that any places where you're using multiple connections, any leaks introduced by clients that forget to Close the connection will also not affect your server.

My $0.02: Don't blindly trust any organisation's (even Google's) ability to design and maintain API. This is a classic case of defaults-gone-wrong.

like image 194
Angad Avatar answered Sep 22 '22 14:09

Angad


One other thing I'm doing is checking the connection to see if it's either open, idle or connecting, and reusing the connection if so. Otherwise, redialing.

grpc will manage your connections for you, reconnecting when needed, so you should never need to monitor it after creating it unless you have very specific needs.

"transport is closing" has many different reasons for happening; please see the relevant question in our FAQ and let us know if you still have questions: https://github.com/grpc/grpc-go#the-rpc-failed-with-error-code--unavailable-desc--transport-is-closing

like image 21
Doug Fawley Avatar answered Sep 20 '22 14:09

Doug Fawley