Menu
I am still starting out with AWS Glue and I am trying to connect it to my publicly accessible MySql database hosted on RDS Aurora to get its data.
So I start by creating a crawler and in the data store I create a new connection as in the screenshot below:
I go through the rest and eventually try to run the crawler but I get the following error: At least one security group must open all ingress ports.To limit traffic, the source security group in your inbound rule can be restricted to the same security group
I am not sure what I need to change in the security group attached to the RDS but here's what I have right now for the inbound rules:
You'll notice that I have a self-referencing rule in there that's pointing to the same security group.
The outbound rules are going to all traffic.
Any idea what I might be doing wrong?
720
Amazon RDS allows you to control access to your DB instances using database security groups (DB security groups). A DB security group acts like a firewall controlling network access to your DB instance. By default, network access is turned off for your DB instances.
AWS Glue uses private IP addresses in the subnet when it creates elastic network interfaces in your specified virtual private cloud (VPC) and subnet. Security groups specified in the connection are applied on each of the elastic network interfaces.
To set up a security groupSign in to the AWS Management Console and open the Amazon EC2 console at https://console.aws.amazon.com/ec2/ . In the left navigation pane, choose Security Groups. Either choose an existing security group from the list, or Create Security Group to use with your notebook server.
The inbound rule (Glue Connection security group) is set to allow TCP Port 0 to allow traffic. Instead, it should allow ALL traffic. Edit your rules, and where there's a dropdown that says "Custom TCP Rule", and change it to "All TCP".
The documentation explains how to setup the security group
190
To solve the second error mentioned above in the comments (VPC S3 endpoint validation failed for SubnetId: subnet-1944ab40. VPC: vpc-c8605bad. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-1944ab40 in Vpc vpc-c8605bad) you have to create an Amazon VPC Endpoints for Amazon S3. https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html
24
You need to set a new rule in the security group that is attached to your DB instances where you define:
21
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
