Can't get real user's IP from X-Forwarded-For

Question

I'm running Traefik 1.7.3 on a single node Kubernetes cluster and I'm trying to get the real user IP from the X-Forwarded-For header but what I get instead is X-Forwarded-For: 10.244.0.1 which is an IP in my k8s cluster.

Here's my Traefik deployment and service:

    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: traefik-conf
    data:
      traefik.toml: |
        # traefik.toml
        debug = true
        logLevel = "DEBUG"
        defaultEntryPoints = ["http","https"]
        [entryPoints]
          [entryPoints.http]
          address = ":80"
          compress = true
          [entryPoints.http.forwardedHeaders]
          trustedIPs = [ "0.0.0.0/0" ]
          entryPoint = "https"
          [entryPoints.https]
          address = ":443"
          compress = true
          [entryPoints.https.forwardedHeaders]
          trustedIPs = [ "0.0.0.0/0" ]
          [entryPoints.https.tls]

        [acme]
        email = "xxxx"
        storage = "/acme/acme.json"
        entryPoint = "https"
        onHostRule = true
        #caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
        acmeLogging = true
        [[acme.domains]]
        main = "xxxx"
        [acme.dnsChallenge]
        provider = "route53"
        delayBeforeCheck = 0

        [persistence]
        enabled = true
        existingClaim = "pvc0"
        annotations = {}
        accessMode = "ReadWriteOnce"
        size = "1Gi"

        [kubernetes]
        namespaces = ["default"]
        [accessLog]
        filePath = "/acme/access.log"
        [accessLog.fields]
        defaultMode = "keep"
    ---
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: traefik-ingress-controller
      namespace: default
      labels:
        k8s-app: traefik-ingress-lb
    spec:
      replicas: 1
      selector:
        matchLabels:
          k8s-app: traefik-ingress-lb
      template:
        metadata:
          labels:
            k8s-app: traefik-ingress-lb
            name: traefik-ingress-lb
        spec:
          serviceAccountName: traefik-ingress-controller
          terminationGracePeriodSeconds: 60
          containers:
          - image: traefik
            name: traefik-ingress-lb
            env:
              - name: AWS_ACCESS_KEY_ID
                value: xxxx
              - name: AWS_SECRET_ACCESS_KEY
                value: xxxx
              - name: AWS_REGION
                value: us-west-2
              - name: AWS_HOSTED_ZONE_ID
                value: xxxx
            ports:
            - name: http
              containerPort: 80
            - name: admin
              containerPort: 8080
            args:
            - --api
            - --kubernetes
            - --configfile=/config/traefik.toml
            volumeMounts:
            - mountPath: /config
              name: config
            - mountPath: /acme
              name: acme
          volumes:
          - name: config
            configMap:
              name: traefik-conf
          - name: acme
            persistentVolumeClaim:
              claimName: "pvc0"
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: traefik-ingress-service
      namespace: default
    spec:
      externalIPs:
      - x.x.x.x
      externalTrafficPolicy: Local
      selector:
        k8s-app: traefik-ingress-lb
      ports:
        - protocol: TCP
          port: 80
          name: web
        - protocol: TCP
          port: 443
          name: https
        - protocol: TCP
          port: 8080
          name: admin
      type: NodePort

And here's my ingress:

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: headers-test
      namespace: default
      annotations:
        ingress.kubernetes.io/proxy-body-size: 500m
        kubernetes.io/ingress.class: traefik
    spec:
      rules:
      - host: xxxx
        http:
          paths:
          - path: /
            backend:
              serviceName: headers-test
              servicePort: 8080

I'd read that I only needed to add [entryPoints.http.forwardedHeaders] and a list of trustedIPs but that doesn't seem to work. Am I missing something?

Answer 1

If you use NodePort for the Traefik Ingress Service, you will have to set service.spec.externalTrafficPolicy to "Local". Otherwise you will have a SNAT when your connection enters the K8s-cluster. This SNAT is necessary to forward the incoming connection to your pod if it is not running on the same node.

But be aware that having set service.spec.externalTrafficPolicy to "Local", only the node on which the Traefik pod is executed will accept requests on 80, 443, 8080. There is no forwarding to the pod from the other nodes anymore. This can result in odd delays when connecting to your service. To avoid that your Traefik would need to run in a HA setup (DaemonSet). Just keep in mind that you need a K/V-Store for a distributed Traefik setup to make Letsencrypt work well.

If the service.spec.externalTrafficPolicy setting does not yet resolve your problem you might also need to configure the kubernetes overlay network to not do any SNAT.

service.spec.externalTrafficPolicy is nicely explained here: https://kubernetes.io/docs/tutorials/services/source-ip/