How to avoid the automatic deleting of inactive topics in Apache Pulsar

Question

I have an application that produces messages to Pulsar under a specific topic and shut down the application when it's finished; at the same time, no consumer exists to read this topic.

After a while, when I create a consumer and want to read the written data out, I found all data are lost since the topic I've written been deleted by Pulsar.

How can I disable the auto-deletion of inactive topics in Pulsar?

Answer 1

Generally, there are two ways to achieve this.

• Firstly, retention policies keep the data for at least X hours (until Y GBs), you could set it via pulsar-admin to infinite at the namespace level.

  pulsar-admin namespaces set-retention my-tenant/my-ns \
  --size 1T \
  --time -1

• Secondly, manually set brokerDeleteInactiveTopicsEnabled=false in conf/broker.conf could disable the deletion of inactive topics as well.

It's recommended to set the above two settings simultaneously for proper control.

Answer 2

If you create a subscription on the topic using the Consumer client interface or the REST API, messages are kept until they are acknowledged. An unacknowledged message in a subscription backlog will never be removed, unless you configure a time-to-live (TTL) which will automatically acknowledge the message after some time.

Messages that are not in a subscription or have already been acknowledged are retained in the topic based on the retention policy. You can specify messages to be retained by size or time.

If you want to use a Pulsar topic like a queue that holds all messages until they are acknowledged, which sounds like what you are trying to do, you just need to use the Consumer client interface with a named subscription. Then all your messages will be kept in the topic while your application is inactive. And because the topic still has messages, it won't be automatically deleted (though you can disable that behavior as explained in the answer by yjshen).