Menu
For testing purpose, I need to simulate client for generating 100,000 messages per second and send them to kafka topic. Is there any tool or way that can help me generate these random messages?
910
Why can Apache Kafka be used for video streaming? High throughput – Kafka handles large volume and high-velocity data with very little hardware. It also supports message throughput of thousands of messages per second. Low Latency – Kafka handles messages with very low latency in the range of milliseconds.
Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in an Apache Kafka® cluster. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster technology.
Apache Kafka is used for both real-time and batch data processing, and is the chosen event log technology for Amadeus microservice-based streaming applications. Kafka is also used for operational use cases such as application logs collection.
Kafka has a variety of use cases, one of which is to build data pipelines or applications that handle streaming events and/or processing of batch data in real time.
There's a built-in tool for generating dummy load, located in bin/kafka-producer-perf-test.sh (https://github.com/apache/kafka/blob/trunk/bin/kafka-producer-perf-test.sh). You may refer to https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java#L106 to figure out how to use it.
One usage example would be like that:
bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000 --throughput 100000
The key here is the --throughput 100000 flag which indicated "throttle maximum message amount to approx. 100000 messages per second"
139
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
