KAFKA broker throwing Wrong request type 18 and Wrong request type 16

Question

System:
EC2 Instance Type: m5.2xlarge
OS: Ubuntu 14.04.5 LTS
Kernel Version: 4.4.0-1022-aws
vCPU: 8 Cores
Memory: 32Gib

Java Version: 1.8.0_171

We are running KAFKA in cluster mode with two kafka brokers(10.0.51.1 & 10.0.51.2) and three zookeeper nodes. I wanted to upgrade my AWS EC2 instance, so I ran sudo apt-get update and sudo apt-get upgrade and installed linux-aws kernel. After changing instance type, I started getting following error in my 10.0.51.2 kafka borker:

18/06/08 07:03:50 52902420 [kafka-network-thread-9092-0] ERROR kafka.network.Processor - Closing socket for /10.0.51.1 because of error
kafka.common.KafkaException: Wrong request type 18
    at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:64)
    at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:50)
    at kafka.network.Processor.read(SocketServer.scala:450)
    at kafka.network.Processor.run(SocketServer.scala:340)
    at java.lang.Thread.run(Thread.java:748)
18/06/08 07:03:50 52902571 [kafka-network-thread-9092-2] ERROR kafka.network.Processor - Closing socket for /10.0.51.1 because of error
kafka.common.KafkaException: Wrong request type 16
    at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:64)
    at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:50)
    at kafka.network.Processor.read(SocketServer.scala:450)
    at kafka.network.Processor.run(SocketServer.scala:340)
    at java.lang.Thread.run(Thread.java:748)

Only 1 broker is giving this error. But my KAFKA cluster is up and running, I can consume and produce from all topics.

Any help would be appreciated.

Thanks.

Answer 1

The client sent an ApiVersionsRequest (request code 18) and a ListGroupsRequest (request code 16). see source code

Probably not all brokers/consumers/producers are using the same version after the upgrade. Can you please verify and in case align the versions? new clients should be able to communicate to older version so probably the library used in your code is older.