Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using docker GELF driver env/labels in logstash

Docker GELF log driver allows env and labels log-opts:

The labels and env options are supported by the gelf logging driver. It adds additional key on the extra fields, prefixed by an underscore (_) (ref)

I want to use this in my index name for elasticsearch output but I couldn't figure out how I can access these value or said extra fields.

Assuming that I have these options running a container,

docker run -it \
  --log-driver gelf \
  --log-opt gelf-address=udp://127.0.0.1:12201 \
  --log-opt tag=some-app \
  --log-opt env=staging \
  --log-opt labels=staging \
  ubuntu:16.04 /bin/bash -c 'echo Hello World'

I'd like to use the env value that I passed in my logstash config as such:

input {
  gelf { }
}

output {
  elasticsearch {
    hosts => ["http://127.0.0.1:9200"]
    index => "logstash-%{env-value-here}-%{tag}-%{+YYYY.MM.dd}"
  }
}

There seems to be another question about env/labels with Graylog: Docker GELF driver env option

like image 433
Eren Güven Avatar asked Oct 21 '16 14:10

Eren Güven


People also ask

What is logstash GELF?

logstash-gelf provides a HTTP/HTTPS transport to send log events to HTTP endpoints. The HTTP sender uses POST to send uncompressed JSON data. It sets the Content-type header to application/json and expects response status 202 Accepted .

How do you pull the logs from Docker container to the elk and describe the process?

A typical ELK pipeline in a Dockerized environment looks as follows: Logs are pulled from the various Docker containers and hosts by Logstash, the stack's workhorse that applies filters to parse the logs better. Logstash forwards the logs to Elasticsearch for indexing, and Kibana analyzes and visualizes the data.

What is GELF driver?

The gelf logging driver is a convenient format that is understood by a number of tools such as Graylog, Logstash, and Fluentd. Many tools use this format. In GELF, every log message is a dict with the following fields: version. host (who sent the message in the first place)

How do I run a container using json-file driver?

To use the json-file driver as the default logging driver, set the log-driver and log-opts keys to appropriate values in the daemon. json file, which is located in /etc/docker/ on Linux hosts or C:\ProgramData\docker\config\ on Windows Server. If the file does not exist, create it first.


1 Answers

After reading the PR that added this option, I realised that I misunderstood how it was supposed to work.

--log-opt labels=a,b,c (same with env) define keys to include in the GELF event. The values are actually retrieved from docker labels and environment variables respectively.

--log-opt labels=foo --label foo=bar will include foo: bar in the event.

like image 184
Eren Güven Avatar answered Oct 19 '22 11:10

Eren Güven