Traefik Bad Gateway

Question

I've got some strange issue. I have following setup: one docker-host running traefik as LB serving multiple sites. sites are most php/apache. HTTPS is managed by traefik. Each site is started using a docker-compose YAML containing the following:

version: '2.3'
services:
  redis:
    image: redis:alpine
    container_name: ${PROJECT}-redis
    networks:
      - internal
  php:
    image: registry.gitlab.com/OUR_NAMESPACE/docker/php:${PHP_IMAGE_TAG}
    environment:
      - APACHE_DOCUMENT_ROOT=${APACHE_DOCUMENT_ROOT}
    container_name: ${PROJECT}-php-fpm
    volumes:
       - ${PROJECT_PATH}:/var/www/html:cached
       - .docker/php/php-ini-overrides.ini:/usr/local/etc/php/conf.d/99-overrides.ini
    ports:
      - 80
    networks:
      - proxy
      - internal
    labels:
      - traefik.enable=true
      - traefik.port=80
      - traefik.frontend.headers.SSLRedirect=false
      - traefik.frontend.rule=Host:${PROJECT}
      - "traefik.docker.network=proxy"

networks:
  proxy:
    external:
      name: proxy
  internal:

(as PHP we use 5.6.33-apache-jessie or 7.1.12-apache f.e.)

Additionally to above, some sites get following labels:

traefik.docker.network=proxy
traefik.enable=true
traefik.frontend.headers.SSLRedirect=true
traefik.frontend.rule=Host:example.com, www.example.com
traefik.port=80
traefik.protocol=http

what we get is that some requests end in 502 Bad Gateway traefik debug output shows:

time="2018-03-21T12:20:21Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://172.18.0.8:80, code: 502, Length: 11, duration: 2.516057159s"

can someone help with that? it's completely random when it happens our traefik.toml:

debug = true
checkNewVersion = true
logLevel = "DEBUG"

defaultEntryPoints = ["https", "http"]
[accessLog]

[web]
address = ":8080"

[web.auth.digest]
users = ["admin:traefik:some-encoded-pass"]

[entryPoints]
  [entryPoints.http]
  address = ":80"
#    [entryPoints.http.redirect] # had to disable this because HTTPS must be enable manually (not my decission)
#      entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]


[retry]

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.com"
watch = true
exposedbydefault = false


[acme]
email = "[email protected]"
storage = "acme.json"
entryPoint = "https"
onHostRule = true

[acme.httpChallenge]
entryPoint = "http"

Could the issue be related to using the same docker-compose.yml?

Answer 1

The error "bad gateway" is returned when the web server in the container doesn't allow traffic from traefik e.g. because of wrong interface binding like localhost instead of 0.0.0.0.

Take Ruby on Rails for example. Its web server puma is configured by default like this (see config/puma.rb):

port        ENV.fetch("PORT") { 3000 }

But in order to allow access from traefik puma must bind to 0.0.0.0 like so:

bind "tcp://0.0.0.0:#{ ENV.fetch("PORT") { 3000 } }"

This solved the problem for me.

Answer 2

For anyone getting the same issue:

After recreating the network (proxy) and restarting every site/container it seems to work now. I still don't know where the issue was from.

Answer 3

Another reason can be that you might be accidentally mapping to the vm's port instead of the container port.

I made a change to my port mapping on the docker-compose file and forgot to update the labeled port so it was trying to map to a port on the machine that was not having any process attached to it

Wrong way:

ports:
  - "8080:8081"
labels:
  - "traefik.http.services.front-web.loadbalancer.server.port=8080"

Right way

ports:
  - "8080:8081"
labels:
  - "traefik.http.services.front-web.loadbalancer.server.port=8081"

Also in general don't do this, instead of exposing ports try docker networks they are much better and cleaner. I made my configuration documentation like a year ago and this was more of a beginner mistake on my side but might help someone :)