How to achieve multi-tenancy in the context of Kafka and storm?

Question

What are the best practices, for building a multi-tenant app in the context of Kafka and storm?

For example: creating topic for each tenant and consume multi-topics spout (using wildcard).

Answer 1

I think that topic for each tenant is the right choice.

The naming convention might be something like this: topic_base_name_tenant_id.

The reasons are:

1. It allows flexible configuration for each tenant (like @Sebastian mentioned earlier).
2. Clearer logical separation.

Now let's say that we will use different approach. For example, partition for each tenant. It might be problematic, since:

1. You are limiting the parallelism level to the number of tenants.
2. Adding new tenants, results adding new partition → republish old messages (The default partitioning algorithm is: message_key % partition_size).

Answer 2

Sometimes you may need multiple topics per application. In that case you could follow the following topic naming convention: topic_base_name_tenant_id. So for a given base topic, you will have as many topics as the number of tenants