Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Conecting AWS Lambda to Redshift - Times out after 60 seconds

I created an AWS Lambda function that:

  • logs onto Redshift via JDBC URL
  • runs a query

Locally, using Node, I can successfully connect to the Redshift instance via JDBC, and execute a query.

var conString = "postgresql://USER_NAME:PASSWORD@JDBC_URL”;
var client = new pg.Client(conString);
client.connect(function(err) {   
  if(err) {
            
      console.log('could not connect to redshift', err);
          
  }  
          
// omitted due to above error

However, when I execute the function on AWS Lambda (where it's wrapped in a async#waterfall block), AWS Cloudwatch logs tells me that the AWS Lambda function timed out after 60 seconds.

Any ideas on why my function is not able to connect?

like image 641
Kevin Meredith Avatar asked Feb 01 '15 16:02

Kevin Meredith


People also ask

How do I create a lambda function for Amazon Redshift?

Create a private Amazon Redshift cluster selecting the VPC and subnet group that you just created. 4. Create a new secret for Amazon Redshift with AWS Secrets Manager. Name your secret "redshift". To create a Lambda function that queries your Amazon Redshift cluster, perform the following steps: 1. Open the Lambda console. 2.

How to solve task timed out after x seconds in AWS Lambda?

To solve the "Task timed out after X seconds" error in AWS lambda, you have to: 1 Increase the function's timeout. The default value is 3 seconds , the maximum is 15 minutes. 2 Increase the function's memory. By default it's set to 128 Mb which is way too low and ads onto the function's execution... More ...

Why do I need AWS Lambda for redshift temporary credentials?

Turns out that switching to Redshift temporary credentials, you need AWS Lambda to be able to access Redshift APIs that are not available by default in your VPC. The options are to route either through ENI or through NAT.

Should I use Lambda or Python for redshift cluster?

I have very similar setup, with the difference being our redshift clusters have public ip, for reasons different than lambda handling, and was changed from private to public at later time. I also use python, but shouldn't really matter. If you have aws support in your plan, I would use it.


2 Answers

I find it's either you open your Redshift security group public to all sources, or none. Because a Lambda function isn't running on a fixed address or even a fixed range of IP addresses, which is completely transparent to users (AKA server-less).

I just saw Amazon announced the new Lambda feature to support VPC yesterday. I guess if we can run a Redshift cluster in a VPC, this could solve the problem.

like image 143
piggybox Avatar answered Oct 16 '22 14:10

piggybox


If you are using serverless-framework v1.5.0, you should add:

iamRoleStatements: - Effect: Allow Action: - ec2:CreateNetworkInterface Resource: '*' - Effect: Allow Action: - ec2:DeleteNetworkInterface - ec2:DescribeNetworkInterfaces Resource: 'arn:aws:ec2:${self:provider.region}:*:network-interface/*'

Also should add all securityGroupIds to Inbounds Rules, like below: screenshot 2017-01-09 23 02 33

More info: https://serverless.com/framework/docs/providers/aws/guide/functions/#vpc-configuration

like image 26
Marckaraujo Avatar answered Oct 16 '22 15:10

Marckaraujo