Kafka High-level Consumer error_code=15

Question

When trying to consume from Kafka using the high-level consumer (using a completely new consumer group), the consumer never starts running. When I switch the logging level to debug, I can see the following two lines repeating themselves over and over:

DEBUG [AbstractCoordinator] 09:43:51.192: Sending coordinator request for group CompletelyNewConsumerGroupThatIneverUsedBefore to broker 172.30.3.61:30000 (id: 1043 rack: null)
DEBUG [AbstractCoordinator] 09:43:51.306: Received group coordinator response ClientResponse(receivedTimeMs=1487666631306, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@58f68932, request=RequestSend(header={api_key=10,api_version=0,correlation_id=197,client_id=consumer-1}, body={group_id=CompletelyNewConsumerGroupThatIneverUsedBefore}), createdTimeMs=1487666631192, sendTimeMs=1487666631192), responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}})

All 6 nodes of the cluster consume data properly. Even the old low-level consumer works. Also in Zookeeper there are only the 6 nodes that should be there and their log files look ok.

From googling I found that the error_code=15 means that the Group coordinator is not available. I assume that means Consumer Group coordinator. Is this a callback problem? Why does the high-level consumer not work?

Answer 1

As it turned out, the all partitions of the __consumer_offsets topic were located on dead nodes (nodes that I turned off and that will never come back). I solved the issue by shutting the cluster down, deleting the __consumer_offsets topic from Zookeeper and then starting the cluster again.