How can I use Route 53 as the DNS Challenge for Lets Encrypt in Traefik?

Question

My local domain is home.turtlesystems.co.uk. I am using Traefik on a local Docker Swarm cluster within this domain.

As there is no direct Internet access to the cluster I cannot use the HTTPS challenge for Lets Encrypt so I am attempting to use Route53 as the DNS provider.

I have set up a Zone in Route53 for my home domain, which is a sub domain of turtlesystems.co.uk which I own.

My traefik.toml file looks like:

debug = true

defaultEntryPoints = ["http", "https"]

[entryPoints]
   [entryPoints.http]
   address = ":80"
      [entryPoints.http.redirect]
      entryPoint = "https"
   [entryPoints.https]
   address = ":443"
      [entryPoints.https.tls]


# Enable ACME (Let's Encrypt) automate SSL
[acme]
email = "xxxxxxxxxxxxxxxxxxxx"
storage = "/etc/traefik/acme.json"
dnsProvider = "route53"
entryPoint = "https"
onDemand = true
onHostRule = true
acmeLogging = true

[[acme.domains]]
main = "home.turtlesystems.co.uk"

# Allow access to the Web UI
[web]
address = ":8080"

# Configure how docker will be run
[docker]
endpoint = "unix://var/run/docker.sock"
domain = "traefik"
watch = true
exposedbydefault = false
swarmmode = true

I have created a service for Portainer that has the following Traefik labels:

traefik.port=9000
traefik.docker.network=traefik-net
traefik.frontend.rule=Host:turtle-host-03.home.turtlesystems.co.uk;PathStripPrefix:/portainer
traefik.backend=portainer
traefik.enable=true
traefik.backend.loadbalancer=wrr

As I have acmeLogging enabled in the traefik.toml file I was hoping to get some more information about what is happening or not happening, but I only get the following INFO logs:

reverse_proxy.1.rqebssg613a8@turtle-host-03    | legolog: 2017/12/15 13:16:32 [INFO][home.turtlesystems.co.uk] AuthURL: https://acme-v01.api.letsencrypt.org/acme/authz/z52B_D2iHeITPqT_7K-Z-Y-ieir3VT4l1qGW6tShrd8
reverse_proxy.1.rqebssg613a8@turtle-host-03    | legolog: 2017/12/15 13:16:32 [INFO][turtle-host-03.home.turtlesystems.co.uk] AuthURL: https://acme-v01.api.letsencrypt.org/acme/authz/OxWRpDR3KZm4E0nGngVSRZgF3iE2nhQ3jlNaWtxbd08
reverse_proxy.1.rqebssg613a8@turtle-host-03    | legolog: 2017/12/15 13:16:32 [INFO][home.turtlesystems.co.uk] acme: Could not find solver for: tls-sni-01
reverse_proxy.1.rqebssg613a8@turtle-host-03    | legolog: 2017/12/15 13:16:32 [INFO][home.turtlesystems.co.uk] acme: Trying to solve DNS-01
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:06Z" level=debug msg="Look for provided certificate to validate [turtle-host-03.home.turtlesystems.co.uk]..."
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:06Z" level=debug msg="Look for provided certificate to validate [turtle-host-03.home.turtlesystems.co.uk]..."
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:06Z" level=debug msg="No provided certificate found for domains [turtle-host-03.home.turtlesystems.co.uk], get ACME certificate."
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:06Z" level=debug msg="Challenge GetCertificate turtle-host-03.home.turtlesystems.co.uk"
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:06Z" level=debug msg="No provided certificate found for domains [turtle-host-03.home.turtlesystems.co.uk], get ACME certificate."
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:06Z" level=debug msg="Challenge GetCertificate turtle-host-03.home.turtlesystems.co.uk"
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:07Z" level=debug msg="Look for provided certificate to validate [turtle-host-03.home.turtlesystems.co.uk]..."
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:07Z" level=debug msg="No provided certificate found for domains [turtle-host-03.home.turtlesystems.co.uk], get ACME certificate."
reverse_proxy.1.rqebssg613a8@turtle-host-03    | time="2017-12-15T13:17:07Z" level=debug msg="Challenge GetCertificate turtle-host-03.home.turtlesystems.co.uk"
reverse_proxy.1.rqebssg613a8@turtle-host-03    | legolog: 2017/12/15 13:17:10 [INFO][home.turtlesystems.co.uk] Checking DNS record propagation using [127.0.0.11:53]

As can be seen it is trying to use a DNS challenge, but I am not getting a certificate.

When I first set all this up it did all work, in fact I wrote a blog about it, but now it does not. When I look at my AWS account I can see that the AWS_ACCESS_KEY I have created for this purpose is being used, but nothing seems to be entered into the Zone.

I am passing AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY and AWS_REGION into the Portainer service as environment variables.

Is there more logging I can turn on? Is there anyway to see logs in AWS for Route 53?

Update

After playing around with this I noticed that Traefik is trying to use 127.0.0.11:53 as the DNS server on which to try and check that the TXT record has been created.

I then added --dns and --dns-search to the Traefik service but this did not have any effect on the address that Trafik uses for DNS. Is there another option I can set in Traefik to force this?

Answer 1

1. Go to AWS, Create AIM custom policy Paste the following JSON as the policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
           "Sid": "",
           "Effect": "Allow",
           "Action": [
               "route53:ChangeResourceRecordSets",
               "route53:GetChange",
               "route53:GetChangeDetails",
               "route53:ListHostedZones"
           ],
           "Resource": [
               "*"
           ]
           },
           {
           "Sid": "",
           "Effect": "Allow",
           "Action": [
               "elasticloadbalancing:DescribeLoadBalancers",
               "elasticloadbalancing:SetLoadBalancerListenerSSLCertificate"
           ],
           "Resource": [
               "*"
           ]
           },
           {
           "Sid": "",
           "Effect": "Allow",
           "Action": [
               "iam:ListServerCertificates",
               "iam:GetServerCertificate",
               "iam:UploadServerCertificate"
           ],
           "Resource": [
               "*"
           ]
           }
       ]
   }
   

name the policy "dnsChallenge" (or whatever you like)

2. Create new AIM user and attach above policy

Copy the new user's keys as you'll need to set them as environment variables

3. Go to AWS Route53 and look at the hosted zone. You'll want 2 A records -- for yourdomain.com and *.yourdomain.com both pointing to the static IP of the host running traefik.

Copy down the Hosted zone ID for the domain you are wildcarding.

Define the following environment variables and make sure they are available when traefik starts.

export AWS_ACCESS_KEY_ID=*****************
export AWS_SECRET_ACCESS_KEY=**********************************
export AWS_HOSTED_ZONE_ID=*************

edit traefik.toml

[acme] # Automatically add Let's Encrypt Certificate.
email = "[email protected]"
storage= "acme.json" # Change to fully qualified and exposed path for docker
entryPoint = "https"
OnHostRule = false
acmelogging = true

# caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
caServer = "https://acme-v02.api.letsencrypt.org/directory"

[acme.dnsChallenge]
  provider = "route53"
  delayBeforeCheck = 0

[[acme.domains]]
  main = "*.yourdomain.com"
  sans = ["yourdomain.com"]

From there its a good idea to run it from the command line and watch for the messages..

Answer 2

Adding on bhlowe's answer, I would use a more restricted IAM profile:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:GetChange",
                "route53:ListHostedZonesByName"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/<INSERT_YOUR_HOSTED_ZONE_ID_HERE>"
            ]
        }
    ]
}