IRecordsProcessor processRecords not invoked by KCL library

Question

We performed the following test several times, and we struggle to find a reasonable explanation on why this could be happening:

• We create a consumer, we wait for it to be ready
• We publish two records on the stream the consumer is listening

Our has one shard, and sometimes the consumer doesn't get notified of the records. We use different workerIds, but it might be possible that an application with the same ApplicationName steals the records.

What are the reasons why a KCL consumer never gets records that are just published?

Answer 1

The problem with my code is that once you have created an application for the first time in the KCL, the matching entry used for checkpointing in Dynamo stores the last position of the shard iterator.

When you will restart the application, the shard iterator might be lagging a lot behind the current time, and it could take lot of iterations before you get to the current position where you publish the record.

So for testing purpose you probably need to create a new app every time and to clean your dynamo tables every night