How to run fluentd in docker within the internal network

Question

I have the following configuration in my docker-compose file:

 fluentd:
    build: ./fluentd
    container_name: fluentd
    expose:
    - 24224
    - 24224/udp
    depends_on:
    - "elasticsearch"
    networks:
    -  internal

 public-site:
    build: ./public-site
    container_name: public-site
    depends_on:
    - fluentd
    logging:
      driver: fluentd
      options:
        tag: public-site
    networks:
    -  internal

networks:
  internal:

When I start the app using docker-compose up, then the webserver exists with the error message ERROR: for public-site Cannot start service public-site: failed to initialize logging driver: dial tcp 127.0.0.1:24224: connect: connection refused.

On the other hand, when I publish the ports from fluentd (ports: 24224:24224), it works. The problem is that I don't want to publish those ports on the host, since it bypasses the linux firewall (i.e. it exposes the fluentd port to everyone, see here).

This is confusing, since exposing a port should make it available for every container in the network. I am using an internal network betweem fluentd and the webserver, so I would expect that the exposed ports of fluentd are enough (which isn't the case).

When I connect to the webserver container, I can ping and resolve the fluentd container, so there is a connection. For some reasons however, at startup it won't accept a fluentd config with no published ports.

Answer 1

The communication to 127.0.0.1 is always problematic if you're in a container. I found this explanation in the docs that performs way better than I would do:

To use the fluentd driver as the default logging driver, set the log-driver and log-opt keys to appropriate values in the daemon.json file, which is located in /etc/docker/ on Linux hosts or C:\ProgramData\docker\config\daemon.json on Windows Server. For more about +configuring Docker using daemon.json, see +daemon.json.

The following example sets the log driver to fluentd and sets the fluentd-address option.

 {
   "log-driver": "fluentd",
   "log-opts": {
     "fluentd-address": "fluentd:24224"
   }
 }

src: https://docs.docker.com/config/containers/logging/fluentd/

EDIT: this works until you want to have an application on the host communicating with the dockerized fluentd (then it's a pain)