Is it possible to use stdout as a fluentd source to capture specific logs for write to elasticsearch?

Question

I'm a noob to both fluentd and elasticsearch, and I'm wondering if it's possible for fluentd to capture specific logs (in this case, custom audit logs generated by our apps) from stdout - use stdout as a source - and write them to a specific index in elasticsearch. Many thanks in advance for your replies.

Answer 1

Yes, you could use fluentd's exec input plugin to launch your apps and capture their stdout. Note this means fluentd would be in charge of launching your application which may not be desirable - in that case if the application already writes to log file you can set fluentd up to tail that file, or if your application logs go to something like syslog you can feed off of that. If you are okay with modifying the application you could also have the application write the logs to fluentd using unix sockets or TCP/UDP messages or by posting logs to an HTTP endpoint. See Input Plugin Overview for more details and examples of different options for input.

Fluentd can further filter those logs for just the custom audit logs. This could be accomplished in several ways, based on log contents or other log entry properties or some combination. Additionally, there are additional plugins that can tally certain log entries and only emit after some thresholds are met. Note that you may not even need to parse anything if you are able to tag the audit messages from the start, but if and how to do this will differ upon the input plugins used.

For sending logs to Elasticsearch you can use uken/fluent-plugin-elasticsearch's output plugin using index_name to control the specific index.