Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Does a Kafka Consumer default batch size?

Does Kafka provide a default batch size for reading messages from a topic? I have the following code that is reading messages from a topic.

  while (true) {
        final ConsumerRecords<String, User> consumerRecords =
                consumer.poll(500));
        if (consumerRecords.count() == 0) {
            noRecordsCount++;
            if (noRecordsCount > giveUp) break;
            else continue;
        }
        consumerRecords.forEach(record -> {
            User user = record.value();
            userArray.add(user);
        });

        insertInBatch(user)
        consumer.commitAsync();
    }
    consumer.close();

In the insertInBatch method, I persist data to a database. This method is getting called every 500 records, even though I haven't specified any batch size in creating the Consumer. I don't think there's anything special about the way I'm creating it. Using Avro for the messages, but I don't think that's significant(?)

Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("group.id", "test");
    props.put("auto.commit.enable", "false");
    props.put("auto.offset.reset", "earliest");

    props.put("key.serializer",StringSerializer.class.getName());
    props.put("value.serializer",KafkaAvroDeserializer.class.getName());
    props.put("schema.registry","http://localhost:8081");
like image 884
Jim Avatar asked Aug 06 '26 05:08

Jim


1 Answers

Yes, there's a default max.poll.records

https://kafka.apache.org/documentation/#consumerconfigs

If you are inserting to a database, though, you'd be better off using Kafka Connect than writing a consumer with apparently no error handling (yet?)

like image 70
OneCricketeer Avatar answered Aug 07 '26 19:08

OneCricketeer



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!