Multiple instances of Traefik in different kubernetes namespaces

Question

We're trying to run a kubernetes cluster with three namespaces:

• public contains services accessible to anyone
• internal contains services that should only be accessed by members of staff
• engineering contains services that should only be visible to developers

The internal and engineering namespaces are protected with mutual authentication, each using a different certificate authority.

We're using Traefik as to manage ingresses to these services however, following the kubernetes guide to set up Traefik, grants each instance of Traefik permissions to see ingresses across all namespaces. This means you could use a service in the internal namespace via the Traefik instance running in the public namespace, bypassing the mututal authentication.

We're working around this by setting hosts on the ingresses, but this means ingresses must be defined seperately for each environment (eg, host: engineering.example.com is different to host: engineering.staging.example.com). We'd prefer to leave the host out of the ingress configuration.

In theory, using RBAC we, should be able to restrict what Traefik is allowed to see to just resources in it's own ingresses as suggested in this guide on RBAC.

My understanding is that it still needs ClusterRole permissions such as this:

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch

But using a Role binding instead of a ClusterRole binding will then restrict those permissions to just the ones in the given service accounts namespace. So if the service account is in the engineering namespace:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: engineering

Then the role binding would be:

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: engineering
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller

We then tie the service account to the Traefik deployment with:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: engineering
spec:
  replicas: 1
  template:
    spec:
      serviceAccountName: traefik-ingress-controller
      ...

We also set the namespaces in the config as per the kubernetes configuration guide

[kubernetes]
namespaces = ["engineering"]

However, when Traefik starts we get the error:

E0313 11:15:57.971237 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:engineering:traefik-ingress-controller" cannot list endpoints at the cluster scope: Unknown user "system:serviceaccount:engineering:traefik-ingress-controller"

The Unknown user is confusing as this is obviously binding a ServiceAccount not a user. Additionally, we can see the ServiceAccount has been created via kubectl.

I'm at a bit of a dead end here.

How do I make Traefik only pick up Ingresses in it's own namespace?

Answer 1

This error may occur when Traefik believes that no namespaces were configured; that is, the TOML configuration you outlined

[kubernetes]
namespaces = ["engineering"]

is not becoming effective.

I can think of two reasons:

1. In addition to the TOML configuration file, you are also passing a --kubernetes command-line argument to Traefik (through an args entry in the Deployment manifest). This would disable the namespaces option.
2. The file is not properly mounted into the Deployment, causing the default namespaces value (the empty list) to be effective. To tell whether that's truly the case, we need to see your full ConfigMap and the relevant volume sections of your Deployment manifests.