Kafka design questions - Kafka Connect vs. own consumer/producer

Question

I need to understand when to use Kafka connect vs. own consumer/producer written by developer. We are getting Confluent Platform. Also to achieve fault tolerant design do we have to run the consumer/producer code ( jar file) from all the brokers ?

Answer 1

Kafka connect is typically used to connect external sources to Kafka i.e. to produce/consume to/from external sources from/to Kafka.

Anything that you can do with connector can be done through Producer+Consumer

Readily available Connectors only ease connecting external sources to Kafka without requiring the developer to write the low-level code.

Some points to remember..

1. If the source and sink are both the same Kafka cluster, Connector doesn't make sense
2. If you are doing changed-data-capture (CDC) from a database and push them to Kafka, you can use a Database source connector.
3. Resource constraints: Kafka connect is a separate process. So double check what you can trade-off between resources and ease of development.
4. If you are writing your own connector, it is well and good, unless someone has not already written it. If you are using third-party connectors, you need to check how well they are maintained and/or if support is available.