Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Difference between request.timeout.ms and timeout.ms properties of Kafka producer

Refering the kafka documentation, a Kafka message producer configuration have request.timeout.ms and timeout.ms properties. Reading the description of these two properties, I am not able clearly distinguish between them.

Can anyone explain the difference using small example.

like image 236
vatsal mevada Avatar asked Nov 24 '16 08:11

vatsal mevada


People also ask

How do you handle timeout in Kafka?

To handle the timeout exceptions, the general practice is: Rule out broker side issues. make sure that the topic partitions are fully replicated, and the brokers are not overloaded. Fix host name resolution or network connectivity issues if there are any.

What is Kafka producer linger MS?

linger.ms refers to the time to wait before sending messages out to Kafka. It defaults to 0, which the system interprets as 'send messages as soon as they are ready to be sent'. batch. size refers to the maximum amount of data to be collected before sending the batch.

What is Max Block MS?

max.block.ms is used for producer to block buffer time, serialization time etc. For details look at this one. https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient. Follow this answer to receive notifications.


2 Answers

request.timeout.ms is the timeout configured on the client side. It says that the client is going to wait this much time for the server to respond to a request.

timeout.ms is the timeout configured on the leader in the Kafka cluster. This is the timeout on the server side. For example if you have set the acks setting to all, the server will not respond until all of its followers have sent a response back to the leader. The leader will wait timeout.ms amount of time for all the followers to respond.

So client sends a request to the server (leader). Based on the acks setting, the server will either wait or respond back to the client. timeout.ms is the amount of time the leader waits for its followers whereas request.timeout.ms is the amount of time the client waits for the server(leader).

like image 112
yaswanth Avatar answered Oct 01 '22 10:10

yaswanth


They are both used for the common underlying network client, meaning the max amount of time for awaiting a response from the request, although timeout.ms is marked as 'Deprecated'. Actually, timeout.ms is only used on the producer side, and request.timeout.ms can be defined for both client(including producer and consumer) and server(for replicating threads).

Kafka recommends that user specify request.timeout.ms instead of timeout.ms

like image 33
amethystic Avatar answered Oct 01 '22 10:10

amethystic