Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Lambda times out connecting to RedShift

My Redshift cluster is in a private VPC. I've written the following AWS Lamba in Node.js which should connect to Redshift (dressed down for this question):

'use strict';
console.log('Loading function');

const pg = require('pg');

exports.handler = (event, context, callback) => {
var client = new pg.Client({
    user: 'myuser',
    database: 'mydatabase',
    password: 'mypassword',
    port: 5439,
    host: 'myhost.eu-west-1.redshift.amazonaws.com'
});


    // connect to our database
    console.log('Connecting...');
    client.connect(function (err) {
        if (err) throw err;

        console.log('CONNECTED!!!');

    });

};

I keep getting Task timed out after 60.00 seconds unfortunately. I see in the logs "Connecting...", but never "CONNECTED!!!".

Steps I've taken so far to get this to work:

  • As per Connect Lambda to Redshift in Different Availability Zones I have the Redshift cluster and the Lamba function in the same VPC
  • Also Redshift cluster and the Lamba function are on the same subnet
  • The Redshift cluster and the Lamba function share the same security group
  • Added an inbound rule at the security group of the Redshift cluster as per the suggestion here (https://github.com/awslabs/aws-lambda-redshift-loader/issues/86)
  • The IAM role associated with the Lamba Function has the following policies: AmazonDMSRedshiftS3Role, AmazonRedshiftFullAccess, AWSLambdaBasicExecutionRole, AWSLambdaVPCAccessExecutionRole, AWSLambdaENIManagementAccess scrambled together from this source: http://docs.aws.amazon.com/lambda/latest/dg/vpc.html (I realize I have some overlap here, but figured that it shouldn't matter)
  • Added Elastic IP to the Inbound rules of the Security Group as per an answer from a question listed prior (even if I don't even have a NAT gateway configured in the subnet)
  • I don't have Enhanced VPC Routing enabled because I figured that I don't need it.
  • Even tried it by adding the Inbound rule 0.0.0.0/0 ALL types, ALL protocols, ALL ports in the Security Group (following this question: Accessing Redshift from Lambda - Avoiding the 0.0.0.0/0 Security Group). But same issue!

So, does anyone have any suggestions as to what I should check?

*I should add that I am not a network expert, so perhaps I've made a mistake somewhere.

like image 805
Danny van der Kraan Avatar asked May 22 '17 11:05

Danny van der Kraan


People also ask

Why does my Lambda keep timing out?

There are three reasons why retry and timeout issues occur when invoking a Lambda function with an AWS SDK: A remote API is unreachable or takes too long to respond to an API call. The API call doesn't get a response within the socket timeout.

Can't connect to server connection timed out Lambda?

Make sure that you confirm that there's a valid connection before attempting to use the connection. If there's not a valid connection, then create a new connection before continuing. As a test, launch an Amazon Elastic Compute Cloud (Amazon EC2) instance with the same Amazon VPC configuration as your Lambda function.

What is the most likely reason for the Lambda function timing out when connected with the Lex bot?

There are many reasons why a function might time out, but the most likely is that it was waiting on an IO operation to complete. Maybe it was waiting on another service (such as DynamoDB or Stripe) to respond. Within a Lambda invocation, the function might perform multiple IO operations.


1 Answers

The timeout is probably because your lambda in VPC cannot access Internet in order to connect to your cluster(you seem to be using the public hostname to connect). Your connection options depend on your cluster configuration. Since both your lambda function and cluster are in the same VPC, you should use the private IP of your cluster to connect to it. In your case, I think simply using the private IP should solve your problem.

Depending on whether your cluster is publicly accessible, there are some points to keep in mind.

  • If your cluster is configured to NOT be publicly accessible, you can use the private IP to connect to the cluster from your lambda running in a VPC and it should work.

  • If you have a publicly accessible cluster in a VPC, and you want to connect to it by using the private IP address from within the VPC, make sure the following VPC parameters to true/yes:

    • DNS resolution
    • DNS hostnames

The steps to verify/change these settings are given here.

If you do not set these parameters to true, connections from within VPC will resolve to the EIP instead of the private IP and your lambda won't be able to connect without having Internet access(which will need a NAT gateway or a NAT instance).

Also, an important note from the documentation here.

If you have an existing publicly accessible cluster in a VPC, connections from within the VPC will continue to use the EIP to connect to the cluster even with those parameters set until you resize the cluster. Any new clusters will follow the new behavior of using the private IP address when connecting to the publicly accessible cluster from within the same VPC.

like image 146
user818510 Avatar answered Oct 08 '22 18:10

user818510