Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Amazon Elasticsearch Service: Authorization header issue when calling ES domain via proxy

I am currently migrating our existing ELK application from EC2 to Amazon Elasticsearch Service. Firstly, I decided to keep my existing Kibana server and only switch it to point to the new ES domain. So I had changed my existing kibana configuration, as advised in this section: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-kibana

Then, I started to have some issues, which most of them were addressed in this article: Kibana won't connect to Elasticsearch on Amazon's Elasticsearch Service Except for step 4. Kibana was still requiring authentication via the header. Since I am not using CLI or a AWS SDK to authenticate via access keys, but using HTTP/S calls instead (i.e. http://search-[es-domain]:80 inside the kibana config file), it is looking like I would need to sign my HTTP requests anyway (as suggested in step 4 on the previous link) however, I am looking for other options to overcome this issue, in order to have my own Kibana server, cluster, with the proper level of access control from the outside internet, whilst still protecting access to the Elasticsearch for only certain IAM users/roles.

I then decided to try another option: to implement my own proxy (in this case use my existing Nginx server to make it point to the new ES domain. This approach is suggested in slide 56 on the Reinvent bdt209-launch-amazon-elasticsearch-for-realtime-data-analytics slideshare. That way, I could make the web server accessible to the world (via ports 443/80), with of course basic web authentication, whilst protecting the ES Domain with very restrictive access policy only allowing IP-based (as well as IAM roles) access to the Elasticsearch cluster.

However I still got into the same issue. I'm getting this message as a reponse:

 "message": "'a2liYW5hYWRtaW46YWRtaW5AMTIz' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic a2liYW5hYWRtaW46YWRtaW5AMTIz'."

Which means I would need to sign the request. Can I please have some suggestions on how to overcome this issue? Do I really need to sign the web requests programatically? Or are there other options without affecting security and access control (meaning allowing restricted public access to kibana, with very limited access (role and IP-based) to ES cluster) ?

Thanks very much.

like image 953
tholiv Avatar asked Oct 27 '15 01:10

tholiv


People also ask

Why is my Amazon OpenSearch service domain stuck in the processing state?

Your OpenSearch Service cluster enters the "Processing" state when it's in the middle of a configuration change. The cluster can get stuck in the "Processing" state if either situation occurs: A new set of data nodes fails to launch. Shard migration to the new set of data nodes is unsuccessful.

Is OpenSearch same as Elasticsearch?

The Amazon Elasticsearch Service was renamed to Amazon OpenSearch Service on September 8th 2021 according to the official AWS open-source blog.

What port does AWS Elasticsearch use?

By default, Elasticsearch will use port 9200 for requests and port 9300 for communication between nodes within the cluster.


2 Answers

The root of the issue was Nginx having added the Authorization header (enabled via Basic Authentication module) that includes an encrypted string with the login details to my custom Kibana instance.

If this header is passed to the ES domain endpoint, then ES domain will try to use that header to authenticate against the receiving server. Since it is not preset for my custom authorization credentials, it will simply reject with the Authorization error.

So I had to disable Header Forwarding from the NGINX proxy to the ElasticSearch endpoint replacing my location stanza (in my NGINX config) with this one:

location / {
     proxy_pass http://localhost:5601/;
     proxy_pass_request_headers off;
     proxy_redirect http://localhost:5601/ /;
     proxy_set_header Host $host;
     proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
     auth_basic "Restricted";
     auth_basic_user_file /etc/nginx/nginx.auth;
}
like image 123
tholiv Avatar answered Oct 04 '22 03:10

tholiv


I know there is an approved answer here but it did not completely worked for me and i want to present my solution, maybe someone can use it. My environment is simple, usual aws linux on a t2 instance, apt-get installed nginx and untar kibana folder.

my nginx.conf file ;

server {
        listen 80;
        server_name localhost;
        location / {
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header User-Agent $http_user_agent;
                auth_basic_user_file /etc/nginx/.htpasswd;
                auth_basic "Auth Required";
                proxy_pass http://search-xxxxxx.eu-west-1.es.amazonaws.com/;
                proxy_redirect http://search-xxxxxx.eu-west-1.es.amazonaws.com/ /;

                proxy_set_header Authorization "";
                proxy_hide_header Authorization;

        }
}

I believe last two lines are important and what is missing in other answer. When you send any kind of Authentication header to Aws ES, it tries to find aws api auth method like signature as value, so it does not work with auth header.

I hope this help someone too, i spent 4 hours for that as a nginx newbie.

like image 26
halil Avatar answered Oct 04 '22 02:10

halil