Kafka Client Connection Pooling

Question

Does it make sense to perform producer/consumer connection pooling of kafka clients?

Does kafka internally maintain a list of connection objects initialized and ready to use?

We'd like to minimize time of connection creation, so that there is no additional overhead when it comes to send/receive messages.

Currently we're using apache commons-pool library GenericObjectPool to keep connections around.

Any help will be appreciated.

Answer 1

Kafka clients maintain their own connections to the clusters.

Both the Producer and Consumer keep connections alive to the brokers they are interacting with. In case they stop interacting, after connections.max.idle.ms the connection will be closed. This setting also exists on the broker so you may want to verify with your admin if they changed this value.

So in most cases, once started Kafka clients don't create many new connections but just use the ones created at startup