Logging to logstash from python

Question

I am having some troubles logging to logstash from python.

I am using a docker image from sebp/elk (http://elk-docker.readthedocs.io) and python-logstash (https://pypi.python.org/pypi/python-logstash) as a logging handler

The docker image exposes port 5044 for the Beats interface (http://elk-docker.readthedocs.io/#usage). Whenever I try to log to this port nothing happens... Is there some other port I should be using ?

I have verified that logstash works by doing this:

/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch   { hosts => ["localhost"] } }' --path.data /root/data

Python code (official example from python-logstash):

import logging
from logging import StreamHandler
import logstash
import sys

host = 'localhost'

test_logger = logging.getLogger('python-logstash-logger')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash.TCPLogstashHandler(host, 5044, version=1))
test_logger.addHandler(StreamHandler())

try:
    test_logger.error('python-logstash: test logstash error message.')
    test_logger.info('python-logstash: test logstash info message.')
    test_logger.warning('python-logstash: test logstash warning message.')

    # add extra field to logstash message
    extra = {
        'test_string': 'python version: ' + repr(sys.version_info),
        'test_boolean': True,
        'test_dict': {'a': 1, 'b': 'c'},
        'test_float': 1.23,
        'test_integer': 123,
        'test_list': [1, 2, '3'],
    }
    test_logger.info('python-logstash: test extra fields', extra=extra)
except:
    print("Error")

I have no clue why this is not working. Does it possibly have something to do with Beats?

Answer 1

The real answer here is that python-logstash doesn't use the beats protocol. It uses TCP or UDP.

You can see this in your code logstash.TCPLogstashHandler(host, 5959, version=1)

You need to setup your ELK Docker image to have a TCP or UDP listener and then choose the matching handler in python-logstash to send messages.

Here is an example beats configuration file that you could include in the Docker image to listen on TCP port:

input {
    tcp {
    port => 5959
    codec => json
  }
}

If you named that file 03-tcp-input.conf then your Dockerfile might look like:

FROM sebp/elk

ENV LOGSTASH_PATH_CONF /etc/logstash

ADD ./03-tcp-input.conf ${LOGSTASH_PATH_CONF}/conf.d/03-tcp-input.conf

Study the Docker file in the source repo to understand how Sébastien setup the configuration for Logstash.

Answer 2

It does have something to do with beats, use this docker instead: github.com/deviantony/docker-elk and everything worked like a charm