VPC Endpoint for AWS Secrets Manager

Question

Route table (in private subnet) won't change by adding VPCE as destination for aws secrets manager. Tried with new SG too (not using default SG). Any idea ?

Answer 1

Based on the comments.

Secrets Manager (SM) uses VPC interface endpoints. This is new generation of endpoints, as compared to VPC gateway endpoints for S3 and DynamoDB. The new generation does not modify route tables (RTs). In contrast, the gateway endpoints do modify RTs specified when creating these endpoints.

For seamless work with the interface endpoints, it is important that the VPC has enableDnsHostnames and enableDnsSupport enabled, as well as private DNS for the endpoint. In addition, security group of the endpoint usually needs to be adjusted to allow connections on port 443.

Answer 2

Recently I removed all NAT Gateways which originally allowed my private subnets to connect to the internet (including the Secrets Manager). I assumed that the NAT Gateway was a security vulnerability, expensive infrastructure and not required for private instances to contact the AWS Secrets Manager. I was wrong. My ECS task failed, and stated that the Secrets Manager resource could not be accessed.

---

Debugging

I recommend using the AWS VPC Reachability Analyzer to debug these network issues, it helped me.

1. Launch an EC2 instance in the private subnet
2. Assign your private subnet security groups to the EC2 instance
3. Create a VPC Endpoint for your private subnets
4. Use the VPC Reachability Analyzer to "Create and analyze path"
   • Source Type: Instance
     • Select the new EC2 instance you launched
   • Destination Type: VPC Endpoints
     • Select the VPC Endpoint you created
   • Destination port
     • Set to 443 for HTTPS
   • Protocol
     • Set to TCP
5. Make changes to your infrastructure, and re-run the analyzer to test access until successful.

This will enable you to verify if the EC2 instance you launched in the private subnet can contact the VPC Endpoint (eg. Secrets Manager).

---

Checklist

Now that you have a means to quickly verify reach-ability, here are some steps that you might need to take in order to ensure your private subnet can contact AWS services like the Secrets Manager. Please consult AWS documentation as I may have outdated information.

• Ensure that the AWS::EC2::VPC
  • EnableDnsSupport: true
  • EnableDnsHostnames: true
• Create a AWS::EC2::VPCEndpoint (not free, see pricing)
  • PrivateDnsEnabled: true
  • ServiceName:
    • Use this as a template com.amazonaws.${AWS::Region}.secretsmanager and use the appropriate value for the AWS region
  • SecurityGroupIds:
    • The security group attached to the VPC endpoint must allow incoming connections on port 443 from the private subnet of the VPC
    • This associated the VPC endpoint with a security group used to allow incoming traffic from the private subnet to AWS API
  • SubnetIds:
    • The private subnets
  • VpcEndpointType: Interface
  • VpcId: the VPC to place the VPC endpoint