Distributed Kafka Connect topic configuration

Question

I had the problem that my Kafka Connect worker configuration was lost after a node restart. (http://broker:port/connectors/ -> empty array)

Now I think it can have something to do with the "retention.ms" config. Because the connect config is also stored in the "config.storage.topic" and will be deleted after "retention.ms"? So I must set a very high "retention.ms". Is this correct or is this automatically managed by Kafka? (in case you create the topic yourself)

How about the other two topics: status.storage.topic - only current state info, not that important? offset.storage.topic

Answer 1

Yes, this could be.

The "config.storage.topic", "status.storage.topic" and "offset.storage.topic" should all be configured with 'cleanup.policy=compact' (it will be 'delete' by default). With this policy, the retention time will have no effect - we will always keep the latest configuration for each connector.