AWS Lambda function is reading records from kinesis stream infinitely

Question

I have a kinesis stream with one shard and a lambda function that was written in python. I added the kinesis stream as an event source with batch size 5. I added couple of hundreds of records to kinesis and lambda function got invoked and executed properly. But for last 3 records the lambda function is getting invoked infinitely even though function return is success.

Lambda function:

from __future__ import print_function

import base64
import json
import urllib2
import json

print('Loading function')

def is_valid_url(url):
    try:
        urllib2.urlopen(url)
        print("Valid URL ...")
        return True         # URL Exist
    except ValueError, ex:
        print("URL is not formatted...")
        return False        # URL not well formatted
    except urllib2.URLError, ex:
        print("Invalid URL ...")
        return False        # URL don't seem to be alive


def lambda_handler(event, context):     
    for record in event['Records']:
        # Kinesis data is base64 encoded so decode here
        payload = base64.b64decode(record['kinesis']['data'])
        params = json.loads(payload)
        print("Decoded payload: " + payload  + " : " +     str(is_valid_url(params['url'])) + " : " + str(len(event['Records'])))
    return 'Successfully processed {} records.'.format(len(event['Records']))

````

When I look at the cloud watch logs

  START RequestId: d6033244-1c43-40ea-8886-f38b8c48daa3 Version:   $LATEST 
  Loading function 
  Valid URL ... 
  Decoded payload: { "url": "https://google.com" }
  Valid URL ... 
  Decoded payload: { "url": "https://google.com" }
  Valid URL ... 
  Decoded payload: { "url": "https://google.com" }
  Valid URL ...  
  Decoded payload: { "url": "https://google.com" }
  END RequestId: d6033244-1c43-40ea-8886-f38b8c48daa3 
  REPORT RequestId: d6033244-1c43-40ea-8886-f38b8c48daa3    Duration: 3003.00 ms    Billed Duration: 3000 ms Memory Size: 128 MB    Max Memory Used: 10 MB   
  2016-03-04T17:32:01.030Z d6033244-1c43-40ea-8886-f38b8c48daa3 Task timed out after 3.00 seconds

I couldn't figure out whats happening with the lambda function. Can someone provide some insights into this error.

Answer 1

Because your function is timing out, Lambda treats the run as an error. The error handling policy for Kinesis is to retry the record until it falls off of the Trim Horizon (typically 24 hours), so your function will retry for 24 hours or until it doesn't timeout.

I can't tell, based on what you have posted, why your function is timing out. A quick fix would be to simply increase the timeout value on the Lambda console (under Advanced on the Configuration Tab)