At least one security group must open all ingress ports. AWS Glue connecting to RDS

Question

I am still starting out with AWS Glue and I am trying to connect it to my publicly accessible MySql database hosted on RDS Aurora to get its data.

So I start by creating a crawler and in the data store I create a new connection as in the screenshot below:

I go through the rest and eventually try to run the crawler but I get the following error: At least one security group must open all ingress ports.To limit traffic, the source security group in your inbound rule can be restricted to the same security group

I am not sure what I need to change in the security group attached to the RDS but here's what I have right now for the inbound rules:

You'll notice that I have a self-referencing rule in there that's pointing to the same security group.

The outbound rules are going to all traffic.

Any idea what I might be doing wrong?

Answer 1

The inbound rule (Glue Connection security group) is set to allow TCP Port 0 to allow traffic. Instead, it should allow ALL traffic. Edit your rules, and where there's a dropdown that says "Custom TCP Rule", and change it to "All TCP".

The documentation explains how to setup the security group

Answer 2

To solve the second error mentioned above in the comments (VPC S3 endpoint validation failed for SubnetId: subnet-1944ab40. VPC: vpc-c8605bad. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-1944ab40 in Vpc vpc-c8605bad) you have to create an Amazon VPC Endpoints for Amazon S3. https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html

Answer 3

You need to set a new rule in the security group that is attached to your DB instances where you define:

• Type: All TCP
• Protocol: TCP
• Range: 0 - 65535
• Source: Custom sg-(the id of this/self security group)
• Description: whatever you want