Send bulk of messages Kafka Producer

Question

I'm using Kafka.

I have a list with 10k jsons.

Currently I send the Jsons as follow:

for(int i=0 ;i< jsonList.size(); i++){
     ProducerRecord<K,V> record = new ProducerRecord(topic, jsonList[i]);
     producer.send(record);
}

Send each message.

I want to send the list to kafka and make kafka to send it json after json (not one message with all json strings), Something like:

ProducerRecord<K,V> record = new ProducerRecord(topic, jsonList);
producer.send(record);

How can I do it?

Thanks

Answer 1

Officially by using KafkaProducer and producerRecord you can't do that, but you can do this by configuring some properties in ProducerConfig

batch.size from document producer batch up the records into requests that are sending to same partition and send them at once

The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size.

linger.ms This setting is used for delay time for producer, to hold producer some time so that all request in meantime will be batched up and sent, but batch.size is upper bound on this, if producer get enough batch size it will ignore this property and send batch messages to kafka

The producer groups together any records that arrive in between request transmissions into a single batched request. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting.