Logstash throwing 401 while connecting with AWS Elasticservice

Question

I have AWS Elastic service domain setup, I'm trying to push some data from an ec2 instance to the AWS elasticservice via Logstash.

I have setup the AWS ES domain with Open access policy - Allow all traffic. I have also enabled Fine Grained Control and setup a master user account to access the AWS ES service.

It all works fine with Kibana or regular cURL calls, but logstash fails with a request sending to https://<my_es_hostname>/_license with a 401 response.

I'm not able to figure out why is this call happening. When I try to hit this in browser, I get {"Message":"Your request: '/_license' is not allowed."}

Here is the sample log that I get from logstash:

[INFO ] 2021-06-02 11:40:18.858 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]}
[INFO ] 2021-06-02 11:40:19.902 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/]}}
[WARN ] 2021-06-02 11:40:20.760 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}
[INFO ] 2021-06-02 11:40:21.371 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.10.2) {:es_version=>7}
[WARN ] 2021-06-02 11:40:21.380 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[ERROR] 2021-06-02 11:40:21.443 [[main]-pipeline-manager] elasticsearch - Unable to get license information {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/_license'"}
[ERROR] 2021-06-02 11:40:21.449 [[main]-pipeline-manager] elasticsearch - Could not connect to a compatible version of Elasticsearch {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}

And here is my logstash configuration:

input {
  jdbc {
     jdbc_driver_class => "org.postgresql.Driver"
     jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.20.jar"
     ...
     <other properties to fetch data>
 }
}
output {
  elasticsearch {
    hosts => ["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]
    user => "username"
    password => "password"
    ilm_enabled => false
    index => "my_index"
    document_id => "%{id}"
    doc_as_upsert => true
 }
}

Answer 1

The issue is with Logstash Elasticsearch Output Plugin trying to verify license on the URL <hostname>/_license.

Refer LS should always perform ES license check · Issue #1004 · logstash-plugins/logstash-output-elasticsearch for reported issue/fix.

While that fix gets released, you can follow these steps to get it working:

• Head over to /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch
• Open file license_checker.rb-> Change the method appropriate_license as per the fix suggested in the above github issue. Making method appropriate_license() return true in case of OSS setup.

Answer 2

There is an Opensearch plugin for logstash which solves this.

cd /usr/share/logstash
bin/logstash-plugin install logstash-output-opensearch

Update the Logstash output plugin, replacing elasticsearch with opensearch

output {
    opensearch {
        hosts => "<Your ES Host>"
        ... <OTHER CONFIG>
    }
}