I am currently migrating our existing ELK application from EC2 to Amazon Elasticsearch Service. Firstly, I decided to keep my existing Kibana server and only switch it to point to the new ES domain. So I had changed my existing kibana configuration, as advised in this section: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-kibana
Then, I started to have some issues, which most of them were addressed in this article: Kibana won't connect to Elasticsearch on Amazon's Elasticsearch Service Except for step 4. Kibana was still requiring authentication via the header. Since I am not using CLI or a AWS SDK to authenticate via access keys, but using HTTP/S calls instead (i.e. http://search-[es-domain]:80 inside the kibana config file), it is looking like I would need to sign my HTTP requests anyway (as suggested in step 4 on the previous link) however, I am looking for other options to overcome this issue, in order to have my own Kibana server, cluster, with the proper level of access control from the outside internet, whilst still protecting access to the Elasticsearch for only certain IAM users/roles.
I then decided to try another option: to implement my own proxy (in this case use my existing Nginx server to make it point to the new ES domain. This approach is suggested in slide 56 on the Reinvent bdt209-launch-amazon-elasticsearch-for-realtime-data-analytics slideshare. That way, I could make the web server accessible to the world (via ports 443/80), with of course basic web authentication, whilst protecting the ES Domain with very restrictive access policy only allowing IP-based (as well as IAM roles) access to the Elasticsearch cluster.
However I still got into the same issue. I'm getting this message as a reponse:
"message": "'a2liYW5hYWRtaW46YWRtaW5AMTIz' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic a2liYW5hYWRtaW46YWRtaW5AMTIz'."
Which means I would need to sign the request. Can I please have some suggestions on how to overcome this issue? Do I really need to sign the web requests programatically? Or are there other options without affecting security and access control (meaning allowing restricted public access to kibana, with very limited access (role and IP-based) to ES cluster) ?
Thanks very much.
Your OpenSearch Service cluster enters the "Processing" state when it's in the middle of a configuration change. The cluster can get stuck in the "Processing" state if either situation occurs: A new set of data nodes fails to launch. Shard migration to the new set of data nodes is unsuccessful.
The Amazon Elasticsearch Service was renamed to Amazon OpenSearch Service on September 8th 2021 according to the official AWS open-source blog.
By default, Elasticsearch will use port 9200 for requests and port 9300 for communication between nodes within the cluster.
The root of the issue was Nginx having added the Authorization header (enabled via Basic Authentication module) that includes an encrypted string with the login details to my custom Kibana instance.
If this header is passed to the ES domain endpoint, then ES domain will try to use that header to authenticate against the receiving server. Since it is not preset for my custom authorization credentials, it will simply reject with the Authorization error.
So I had to disable Header Forwarding from the NGINX proxy to the ElasticSearch endpoint replacing my location stanza (in my NGINX config) with this one:
location / {
proxy_pass http://localhost:5601/;
proxy_pass_request_headers off;
proxy_redirect http://localhost:5601/ /;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/nginx.auth;
}
I know there is an approved answer here but it did not completely worked for me and i want to present my solution, maybe someone can use it. My environment is simple, usual aws linux on a t2 instance, apt-get installed nginx and untar kibana folder.
my nginx.conf file ;
server {
listen 80;
server_name localhost;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_set_header User-Agent $http_user_agent;
auth_basic_user_file /etc/nginx/.htpasswd;
auth_basic "Auth Required";
proxy_pass http://search-xxxxxx.eu-west-1.es.amazonaws.com/;
proxy_redirect http://search-xxxxxx.eu-west-1.es.amazonaws.com/ /;
proxy_set_header Authorization "";
proxy_hide_header Authorization;
}
}
I believe last two lines are important and what is missing in other answer. When you send any kind of Authentication header to Aws ES, it tries to find aws api auth method like signature as value, so it does not work with auth header.
I hope this help someone too, i spent 4 hours for that as a nginx newbie.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With