Filebeat and AWS Elasticsearch - Not Working

Question

I have good experience in working with Elasticsearch, I have worked with version 2.4 and now trying to learn new Elasticsearch. I am trying to implement Filebeat to send my apache and system logs to my Elasticsearch endpoint. To save my time I preferred to launch a t2.medium single node instance over AWS Elasticsearch Service under the public domain and I have attached the access policy to allow everyone to access the cluster. The AWS Elasticsearch instance is up and running healthy. I launched a Ubuntu(18.04) server, downloaded the filebeat tar and made the following configuration in filebeat.yml:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["https://my-public-test-domain.ap-southeast-1.es.amazonaws.com:443"]

18.04-  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

I enabled the required modules :

filebeat modules enable system apache

Then as per the filebeat documentation I changed the ownership of the filebeat file and started the filebeat with the following commands :

sudo chown root filebeat.yml 
sudo ./filebeat -e

When I started the filebeat I faced the following permission and ownership issues :

Error loading config from file '/home/ubuntu/beats/filebeat-7.2.0-linux-x86_64/modules.d/system.yml', error invalid config: config file ("/home/ubuntu/beats/filebeat-7.2.0-linux-x86_64/modules.d/system.yml") must be owned by the user identifier (uid=0) or root

To resolve this I changed the ownership for the files which were throwing errors. When I restarted the filebeat service , I started facing the following issue :

Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials

Going through this link , I found that to work with AWS Elasticsearch I will need Beats OSS versions.
So I again downloaded the OSS version for beat from this link and followed the same procedure as above, but still no luck. Now I am facing the following errors :

Error 1:

Attempting to reconnect to backoff(elasticsearch(https://my-public-test-domain.ap-southeast-1.es.amazonaws.com:443)) with 12 reconnect attempt(s)

Error 2:

Failed to connect to backoff(elasticsearch(https://my-public-test-domain.ap-southeast-1.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: This module requires an Elasticsearch plugin that provides the geoip processor. Please visit the Elasticsearch documentation for instructions on how to install this plugin. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"No processor type exists with name [geoip]","header":{"processor_type":"geoip"}}],"type":"parse_exception","reason":"No processor type exists with name [geoip]","header":{"processor_type":"geoip"}},"status":400}

From the second error I can understand that the geoip plugin is not available because of which I facing this error.

What else needs to be done to get this working?
Has anyone been to successfully connect Beats to AWS Elasticsearch?
What other steps I could to take to mitigate the above issue?

Envrionment Details:

• AWS Elasticsearch Version : 6.7
• File Beat : 7.2.0

Answer 1

First, you need to use OSS version of filebeat with AWS ES https://www.elastic.co/downloads/beats/filebeat-oss

Second, AWS ElasticSearch does not provide GeoIP module, so you will need to edit pipelines for any of the default modules you want to use, and make sure GeoIP is removed/commented out.

For example in /usr/share/filebeat/module/system/auth/ingest/pipeline.json (that's the path when installed from deb package - your path will be different of course) comment out:

        {
        "geoip": {
            "field": "source.ip",
            "target_field": "source.geo",
            "ignore_failure": true
        }
    },

Repeat the same for apache module.