How many producers to create in kafka?

Question

In a high volume real time java web app I'm sending messages to apache kafka. Currently I'm sending to a single topic, but in the future I might need to send messages to multiple topics.

In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics?

Here is my code:

props = new Properties();
props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>);
props.put("zk.connectiontimeout.ms", "1000000");
props.put("producer.type", "async");

Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props));

ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("someTopic1", messageTosend);
ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("someTopic2", messageTosend);

producer.send(producerData1);
producer.send(producerData2);

As you can see, once the producer has been created I can use it to send data to different topics. I wonder what is the best practice? If my app sends to multiple topics (each topic gets different data) can/should I use a single producer or should I create multiple producers? When (generaly speaking) should I use more than a single producer?

Answer 1

We have verified in practice that having only one producer is optimal per topic. However, having multiple producers is useful if you encounter the long, fat network problem, in which case we must have multiple connections to fully utilize the network.

Batching and pipelining in a single TCP connection (as is used by Kafka) by itself will not scale to large batches if you must send to a host far away unless you do TCP Tuning to have large window sizes. This is the case when you might experiment with more producers.