Unbuffered bidirectional data streaming with gRPC: how to get the size of the client-side buffer?

Question

I am streaming data from a server to a client and I would like the server not to read and send more data than the client's buffer size.

Given:

service StreamService {
  rpc Stream(stream Buffer) returns (stream Buffer);
}

message Buffer {
  bytes data = 1;
}

My client's program basically looks like:

func ReadFromServer(stream StreamService_StreamClient, buf []byte) (n int, err error) {
  // I actually don't need more than len(buf)...
  // How could I send len(buf) while stream is bidirectional...?
  buffer, err := stream.Recv()
  if err != nil {
     return 0,  err
  }
  n = copy(buf, buffer.Data)
  // buf could also be smaller than buffer.Data...
  return n, nil
}

So how could I send len(buf) while the RPC's stream is bidirectional, i.e. the send direction is used by another independent stream of data? Note that I don't use client or server-side buffering to avoid loosing data when one of them is terminated (my data-source is an I/O).

Answer 1

gRPC provides no mechanism for this. It only provides push-back when a sender needs to slow down. But there will still be buffering happening internally and that is not exposed because gRPC is message-based, not byte-based.

There's really only two options in your case:

1. Server chunks responses arbitrarily. The client Recv()s when necessary and any extra is manually managed for later.
2. The client sends a request asking for a precise amount to be returned, and then waits for the response.

Note that I don't use client or server-side buffering to avoid loosing data when one of them is terminated (my data-source is an I/O).

This isn't how it works. When you do a Send() there is no guarantee it is received when the call returns. When you do a Recv() there is no guarantee that the message was received after the recv call (it could have been received before the call). There is buffering going on, period.