Kafka producer send blocks indefinitely when kafka servers are down

Question

I'm using Kafka 0.11.0.0. I have a test program that publishes to a Kafka topic; if the zookeeper and Kafka servers are down (which is normal in my development environment; I bring them up as needed) then the call to KafkaProducer<>.send() hangs indefinitely.

I either need to have send() return, preferably indicating the error; or I need a way to check whether the servers are up or down. Basically, I want my test tool to be able tell me, "Hey, dummy, start up Kafka!" instead of hanging.

Is there a way for my producer task to determine whether the servers are up or down?

I'm calling the send() like this:

kafkaProducer.send(new ProducerRecord<>(KAFKA_TOPIC, KAFKA_KEY,
    message), (rm, ex) -> {
        System.out.println("**** " + rm + "\n**** " +ex);
});

I have linger.ms = 1; I've tried retries=0, 1, and 2, and send() still blocks. I've never seen the callback called.

Older messages suggest setting metadata.fetch.timeout.ms to a small value, but that's gone in 0.11. Others suggest calling command line utilities to see if the servers are OK...but the referenced utilities also seem to be gone.

What's the graceful way to get this done?

Answer 1

That is strange. It should return with an error saying either "Failed to update metadata" or "Expiring x number of records".

Check request.timeout.ms and max.block.ms setting for your producer. By default request.timeout.ms is 60 seconds long