Setting up RDS (MySQL) database access using IAM to generate access tokens

Question

I have followed the instructions to set up AWS and MySQL such that I should be able to sign in to mysql using mysql-client and a user (named aws_iam) without a password, but with a token generated by awscli with the role attached to my EC2 instance.

The instructions are here

So what I have is:

• An EC2 instance with a role which allows me to generate RDS credentials
• An RDS instance running MySQL, with a user aws_iam which is identified by AWSAuthenticationPlugin
• When signed in to the EC2 instance via SSH I can run mysql -h mydb.randomstring.region.rds.amazonaws.com -u root -p and enter the master password from the RDS set up to get a mysql shell.
• Also when signed in to the EC2 instance via SSH I can run aws rds generate-db-auth-token --hostname mydb.randomstring.region.rds.amazonaws.com --port 3306 --username aws_iam and when running this I get a token that looks like this:

mydb.randomstring.region.rds.amazon.aws.com:port-number/?Action=connect&DBUser=aws_iam&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-Date=current_time&X-Amz-SignedHeaders=host&X-Amz-Security-Token=really-long-url-encoded-string&X-Amz-Credential=string/region/rds-db/aws4_request&X-Amz-Signature=long-hash

Then I run a connection command:

mysql -h mydb.randomstring.region.rds.amazonaws.com --ssl-ca=rds-ca-2015-eu-west-1.pem --ssl-mode=VERIFY_IDENTITY -u aws_iam --enable-cleartext-plugin --password=TOKEN

But then I just get

ERROR 1045 (28000): Access denied for user 'aws_iam'@IP (using password: YES)

A few things I have noticed:

• the "token" is originally url encoded; however decoding it doesn't work either
• the token has lots of parameters in a url format; it is possible only one of these parameters is the actual token but this isn't mentioned in the documentation
• if your EC2 role doesn't have the policy for "rds-db" you can still generate a token. This likely means the token generation doesn't prove that your policy works; but there's no way to debug it other than that

So has anybody enabled this and is there something I have missed?

Answer 1

The documentation does seem sparse.

As preposterous as it seems at first glance, it looks like the whole thing, complete with its url-escaping, is the "authentication token"... you'd just need to enclose it in ' single quotes on the command line.

Here's how I arrived at that conclusion:

My first step trying to work this out was to check the RDS API Reference. There is no GetDbAuthToken action. Curious.

Then, I noticed that aws rds generate-db-auth-token doesn't require a region. How can that be?

Unless...

Reading between the lines, it looks like the operative term here is generate... not synonymous with get (via an API request).

This looks like an entirely local operation, which means it's possible to successfully "generate" an entirely invalid authentication token... exactly the same way that you can generate a pre-signed URL that's syntactically valid but access is still denied, since the request signer lacks the requisite permission.

if your EC2 role doesn't have the policy for "rds-db" you can still generate a token. This likely means the token generation doesn't prove that your policy works.

I'd call this further proof of my assertions, here. In light of what I'm seeing, I'd say successful token generation proves nothing at all.

You're simply generating what they are calling a "token" -- that is, in form, no different than an AWS4-HMAC-SHA256 ("Signature Version 4") signed URL... signed with your credentials.

RDS takes this entire line, and passes it over to IAM using essentially the same mechanism that the external facing service APIs use, to validate it. This also explains why the token is only good for 15 minutes... that's how most of the Query APIs work. Signed requests are only good for +/- 15 minutes. It also explains why they recommend no more than 20 new connections per second -- your RDS instance is making an internal API request to actually validate the token when you try to log in with it.

I'd go back through all the setup steps but use an IAM user instead of an instance role while testing, just to eliminate a little bit of complexity.

The RDS instance's error log may have more information. You may need to set log_warnings to 2 or higher in the parameter group, which is probably a good idea anyway.