AWS kinesis is not returning records

Question

I have following piece of code to fetch

def getSample(accessKey: String, secretKey: String, streamName: String, size: Int, endpoint: String, region: String): Unit = {
        var client=getKinesisClient(accessKey, secretKey, endpoint, region)
        val decoder = Charset.forName("UTF-8").newDecoder()
        var shards = getShards(accessKey, secretKey, streamName, endpoint, region)
        val numberOfShards = shards.iterator()
        var isStop = false
        var sampleRecords = new StringBuilder
        while (numberOfShards.hasNext() && !isStop) {
            val shard = numberOfShards.next()
            var shardIterator: String = null
            val getShardIteratorRequest = new GetShardIteratorRequest()
            getShardIteratorRequest.setStreamName(streamName)
            getShardIteratorRequest.setShardId(shard.getShardId())
            getShardIteratorRequest.setShardIteratorType("TRIM_HORIZON")
            val getShardIteratorResult = client.getShardIterator(getShardIteratorRequest)
            shardIterator = getShardIteratorResult.getShardIterator
            var getRecordsRequest = new GetRecordsRequest
            getRecordsRequest.setShardIterator(shardIterator)
            var records: List[Record] = client.getRecords(getRecordsRequest).getRecords()
            println(records.size())
            var itr = records.iterator()
            var SampleData = new ArrayList[String]()
            while (itr.hasNext()) {
                val record = itr.next()
                val data = decoder.decode(record.getData).toString
                println(data)
            }
        }
    }

This prints that record count is 0. Same i did with aws cli

aws kinesis put-records --records "Data=hello world - hemant,PartitionKey=20150421" --stream-name simpleStream

it returns 0 failed records Then i ran following command

aws kinesis get-shard-iterator --stream-name simpleStream --shard-id 0 --shard-iterator-type TRIM_HORIZON

and

aws kinesis get-records --shard-iterator xxxxxxxxx

It also returns empty array.

Can somebody please help me ? Thanks

Answer 1

GetRecords does not always return records even if there are records in the shard. This is described in FAQ.

https://aws.amazon.com/kinesis/streams/faqs/

Q: Why does GetRecords call return empty result while there is data within my Amazon Kinesis stream?

You're requesting records from TRIM_HORIZON position, that is the oldest position in the shard. If you call get-records many times in a row(10 or 20 or more) and proceed the iterator point, you'll encounter records.

Following forum is also useful:: https://forums.aws.amazon.com/thread.jspa?messageID=509980