unable to proxy from nginx to kibana

Question

I am trying to proxy requests from nginx to kibana (logstash). I can access the kibana dashboard on port 9292 - I can confirm that a service is listening on port 9292. I can successfully proxy from nginx to other services but the proxy directive for kibana (port 9292) does not work - I can proxy to 9200 for elasticsearch. Any ideas on how to troubleshoot this further would be appreciated.

Update: I have tried changing the server setup in upstream to point to 0.0.0.0 as well as the server address but neither option works. The request gets routed to the default server.

Another Update: I have noticed that removing the proxy parameters from the nginx default file allows me to forward the request to the kibana listneing port - however, kibana complains about missing "dashboards/default.json" which I am guessing is due to some missing or misconfigured setup in nginx.

default (/etc/nginx/sites-available)

upstream logstash {
        server 127.0.0.1:9292;  ##kibana
        keepalive 100;
}

server {
        listen 84;
        listen [::]:84 ipv6only=on;
        root /var/www/;
        index index.html index.htm;
        server_name logstash;

        ##logging per server
        access_log /var/log/nginx/logstash/access.log;
        error_log /var/log/nginx/logstash/error.log;

        location / {
                proxy_redirect off;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $host;
                proxy_pass http://logstash;
        }
}

Answer 1

The problem seems to be

proxy_pass http://your-logstash-host;

If you look at the logs in your LogStash Web, you'll see "WARN -- : attack prevented by Rack::Protection::JsonCsrf"

There's some built-in security, which I'm not familiar with, provided by rack-protection to prevent Cross-origin resource sharing attacks. The problem is that the proxy_pass from Nginx looks like a CORS attack to ruby rack protection.

EDIT:

As previously stated, the module Rack::Protection::CSRF is the one throwing this warning.

I have opened the code and we can clearly see what's going on:

def has_vector?(request, headers)
  return false if request.xhr?
  return false unless headers['Content-Type'].to_s.split(';', 2).first =~ /^\s*application\/json\s*$/
  origin(request.env).nil? and referrer(request.env) != request.host
end

So here's the required nginx config required to pass the requests so that Sinatra will accept them:

server {
    listen       80;
    server_name  logstash.frontend.domain.org;

    location / {
        # Proxying all requests from logstash.frontend to logstash.backend
        proxy_pass   http://logstash.backend.domain.org:9292;
        proxy_set_header X-Real-IP $remote_addr;

        # Set Referer and Host to prevent CSRF panick by Sinatra
        proxy_set_header Referer my-host-04;
        proxy_set_header Host my-host-04.domain.org;

        # Alternatively to setting the Referer and Host, you could set X-Requested-With
        #proxy_set_header X-Requested-With XMLHttpRequest;
    }
}