Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Secrets manager extremely slow in EC2s via awscli and boto3

I'm writing a flask API in pycharm. When I run my code locally, requests using boto3 to get secrets from secrets manager take less than a second. However, when I put my code on an EC2, it takes about 3 minutes (tried in both t2.micro and m5.large).

At first I thought it could be a Python issue, so I ran it in my EC2s through the awscli using:

aws secretsmanager get-secret-value --secret-id secretname

It sill took about 3 minutes. Why does this happen? Shouldn't this in theory be faster in an EC2 than in my local machine?

EDIT: This only happens when the EC2 is inside a VPC different than the default VPC.

like image 778
rodrigocf Avatar asked May 06 '18 19:05

rodrigocf


People also ask

Is AWS secrets Manager down?

AWS Secrets Manager Status is Operational.

How do I get rid of AWS Secret immediately?

Resolution. Run the DeleteSecret API call with the ForceDeleteWithoutRecovery parameter to delete the secret permanently. Notes: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

How AWS secrets Manager uses AWS kms?

Secrets Manager integrates with AWS Key Management Service (AWS KMS) to encrypt every version of every secret value with a unique data key that is protected by an AWS KMS key. This integration protects your secrets under encryption keys that never leave AWS KMS unencrypted.

Is Amazon Secrets Manager dependency injection?

You can add Secrets Manager client-side caching library to your projects either directly or through dependency injection. The dependency package is also available through NuGet.


1 Answers

After fighting with this same issue on our local machines for almost two months, we finally had some forward progress today.

It turns out the problem is related to IPv6.

If you're using IPv6, then the secrets manager domain will resolve to an IPv6 address. For some reason the cli is unable to make a secure connection using IPv6. After it times out, the cli falls back to IPv4 and then it succeeds.

To verify if you're resolving to an IPv6 address, just ping secretsmanager.us-east-1.amazonaws.com. Don't worry about the ping response, you just want to see the IP address the domain resolves to.

To fix this problem, you now have 3 options:

  1. Figure out your networking issues. This could be something on your machine or router. If in an AWS VPC, check your routing tables and security groups. Make sure you allow outbound IPv6 traffic (::/0).
  2. Reduce the cli connect timeout to make the IPv6 call fail faster. This will make the IPv4 fallback happen sooner. You may want give a better timeout value, but the general idea is to add something like this: --cli-connect-timeout 1
  3. Disable IPv6. You can either disable IPv6 on your machine/router altogether, or you can adjust your machine to prefer IPv4 for this specific address (See: https://superuser.com/questions/436574/ipv4-vs-ipv6-priority-in-windows-7).

Ultimately, option 1 is the real solution, but since it is so broad, the others might be easier.

Hopefully this helps someone else maintain a bit of sanity when they hit this.

like image 135
LTAcosta Avatar answered Nov 07 '22 11:11

LTAcosta