What is the difference between call_cq and notification_cq in gRPC?

Question

https://github.com/grpc/grpc/blob/master/examples/cpp/helloworld/greeter_async_server.cc#L91

        service_->RequestSayHello(&ctx_, &request_, &responder_, cq_, cq_,
                                  this);

The two occurrences of cq_ look strange to me so I dig into the source code which leads me to

https://github.com/grpc/grpc/blob/master/include/grpcpp/impl/codegen/service_type.h#L92

  void RequestAsyncUnary(int index, ServerContext* context, Message* request,
                         internal::ServerAsyncStreamingInterface* stream,
                         CompletionQueue* call_cq,
                         ServerCompletionQueue* notification_cq, void* tag) {
    server_->RequestAsyncCall(methods_[index].get(), context, stream, call_cq,
                              notification_cq, tag, request);
  }

So what's the difference between call_cq and notification_cq? What are the potential uses/benefits for using difference completion queues?

Answer 1

Here's a quote from the google-groups forum for grpc when this same question was asked. https://groups.google.com/forum/#!topic/grpc-io/V4NAQ77PMEo

Notification_cq gets the tag back indicating a call has started. All subsequent operations (reads, writes, etc) on that call report back to call_cq. For most async servers my recommendation is to use the same cq. Places where you might not:

1. Our sync API creates a cq per call under the covers... So it posts a general event >queue for notification_cq, and it's specific queue as call_cq.
2. If you want to be able to control when you accept incoming calls vs when you don't (by suspending polling on a notification_cq)
3. I'm sure folks can think of others.

This allows fine-grained control over which threads handle which kinds of events (based on which queues they are polling). Like you may have a master thread polling the notification_cq and worker threads all polling their own call_cqs, or something like that.