Menu
I am currently sending data using spring-kafka like this:
val json = objectWriter.writeValueAsString(obj)
kafkaTemplate.send(topic, json)
How do i tell KafkaTemplate to use compress the json using snappy before sending?
684
To make Kafka compression more effective, use batching. Kafka producers internally use a batching mechanism to send multiple messages in one batch over the network.
Message Compression is always done at the producer side, so there is no requirement to change the configurations at the consumer or broker side. In the figure, a producer batch of 200 MB is created. After compression, it is reduced to 101 MB. To compress the data, a 'compression.
Kafka supports 4 compression codecs: none , gzip , lz4 and snappy .
size measures batch size in total bytes instead of the number of messages. It controls how many bytes of data to collect before sending messages to the Kafka broker. Set this as high as possible, without exceeding available memory. The default value is 16384.
In apache kafka there is producer config compression.type with valid values
The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd
So you can set in producer configs
ProducerConfig.COMPRESSION_TYPE_CONFIG "snappy"
or by using properties
spring.kafka.producer.compression-type= # Compression type for all data generated by the producer.
1.) Compression at broker level.
You can start your zookeeper server with --config compression.type=gzip at the end of the command or you can add this property in zookeeper config file.
2.) Compression at producer level
Set the property compression.type = gzip in config/producer.properties of your producer or start your producer with below property
--compression-codec 'gzip', 'snappy', 'lz4', or 'zstd'.
If specified without value, then it
defaults to 'gzip'
33
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
