How to Send On Premises Kubernetes Logs to Stackdriver

Question

Objective: Get some logging/monitoring on Googles Stackdriver from a Kuberntes HA cluster that is on premises, version 1.11.2.

I have been able to send logs to Elasticsearch using Fluentd Daemonset for Kubernetes, but the project is not supporting Stackdriver (issue). That said, there is a docker image created for Stackdriver (source), but it does not have the daemonset. Looking at other daemonsets in this repository, there are similarities between the different fluent.conf files with the exception of the Stackdriver fluent.conf file that is missing any environment variables.

As noted in the GitHub issue mentioned above there is a plugin located in the Kubernetes GitHub here, but it is legacy. The docs can be found here.

It states:

"Warning: The Stackdriver logging daemon has known issues on platforms other than Google Kubernetes Engine. Proceed at your own risk."

Installing in this manner fails, without indication of why.

Some other notes. There is Stackdriver Kubernetes Monitoring that clearly states:

"Easy to get started on any cloud or on-prem"

on the front page, but doesn't seem to explain how. This Stack Overflow question has someone looking to add the monitoring to his AWS cluster. It seems that it is not yet supported.

Furthermore, on the actual Google Stackdriver it is also stated that

"Works with multiple clouds and on-premises infrastructure".

Of note, I am new to Fluentd and the Google Cloud Platform, but am pretty familiar with administering an on-premise Kubernetes cluster.

Has anyone been able to get monitoring or logging to work on GCP from another platform? If so, what method was used?

Answer 1

Consider reviewing this documentation for using the BindPlane managed fluentd service from Google partner Blue Medora. It is available in Alpha to all Stackdriver users. It parses/forwards Kubernetes logs to Stackdriver, with additional payload markup. Disclaimer: I am employed by Blue Medora.