Menu
I have been trying to find a way to include a filter that discards the records with a particular value, in the debezium source sql connector. Lets say, I have the JSON: {"id":0, "name":"muj"} and I want to discard all the records that have the field "name" value as "muj". Is there a way to do that without having to use confluent platform?
614
Summary. Kafka is unable to perform flexible or fine-grained event filtering due to its simple event routing & event storage per partition architecture.
The Kafka Connect JDBC Sink connector allows you to export data from Apache Kafka® topics to any relational database with a JDBC driver. This connector can support a wide variety of databases. The connector polls data from Kafka to write to the database based on the topics subscription.
Kafka Connect in distributed mode uses Kafka itself to persist the offsets of any source connectors. This is a great way to do things as it means that you can easily add more workers, rebuild existing ones, etc without having to worry about where the state is persisted.
Source connectors can also collect metrics from all your application servers and store the data in Kafka topics–making the data available for stream processing with low latency.
You could use the filter function of Kafka Connect transformations (the one from Confluent). The filter function allows to include or exclude records that match the predicate based on record values. Please, see transforms/filter documentation and examples.
126
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
