How to decide total number of partition keys in AWS kinesis stream?

Question

In a producer-consumer web application, what should be the thought process to create a partition key for a kinesis stream shard. Suppose, I have a kinesis stream with 16 shards, how many partition keys should I create? Is it really dependent on the number of shards?

Answer 1

Partition (or Hash) Key: starts from 1 up to 340282366920938463463374607431768211455. Lets say ~34020 * 10^34, I will omit 10^34 for ease...

If you have 30 shards, uniformly divided, each should cover 1134 * 10^34 hash keys. The coverage should be like this.

Shard-00: 0 - 1134 Shard-01: 1135 - 2268 Shard-03: 2269 - 3402 Shard-04: 3403 - 4536 ... Shard-28: 30619 - 31752 Shard-29: 31753 - 32886 Shard-30: 32887 - 34020

And if you have 3 consumer applications (listening to these 30 shards) each should listen 10 shards (optimum balanced).

This also explains Merge and Split operations on a Stream.

• To merge 2 shards, they should cover adjacent hash keys. You cannot merge Shard-03 and Shard-29.
• You can split any shard. If you split shard-00 in the middle, the distribution will like this;

Shard-31: 0 - 567 Shard-32: 568 - 1134 Shard-01: 1135 - 2268 Shard-03: 2269 - 3402 Shard-04: 3403 - 4536 ... Shard-28: 30619 - 31752 Shard-29: 31753 - 32886 Shard-30: 32887 - 34020

See, Shard-00 will no longer accept new data. The new records that are put in Kinesis stream with the same partition key range (as Shard-00) will be placed under Shard-31 or Shard-32.

While sending data to Kinesis (ie. producer side), you should not worry about "which shard the data goes to". Sending a random number (or uuid, or current timestamp in millis) would be best for scaling and distributing the data effectively on shards. Unless you are worried about the ordering of records in a single shard, it is best to choose a random number/constantly changing partition key for put_record request.

In Java you can use "putRecordsRequestEntry.setPartitionKey(Long.toString(System.currentTimeMillis()))" or "putRecordRequest.setPartitionKey(Long.toString(System.currentTimeMillis()))" can be examples.