Debugging 502 Bad Gateway error - Ubuntu, Nginx, Unicorn

Question

I have an EC2 instance, trying to run a Rails app with Nginx and Unicorn.

I keep getting a 502 Bad Gateway error after about 60 seconds:

Here's what I get in the nginx error log:

2017/02/10 18:03:38 [error] 13208#13208: *1 upstream prematurely closed connection while reading response header from upstream, client: 174.25.146.24, server: _, request: "GET /welcome HTTP/1.1", upstream: "http://unix:/home/rails/myapp/shared/sockets/unicorn.sock:/welcome", host: "mydomain.com"

unicorn.stderr.log

E, [2017-02-13T00:58:12.456504 #13149] ERROR -- : worker=0 PID:16535 timeout (31s > 30s), killing E, [2017-02-13T00:58:12.459388 #13149] ERROR -- : reaped # worker=0 I, [2017-02-13T00:58:12.459499 #13149] INFO -- : worker=0 spawning... I, [2017-02-13T00:58:12.461390 #16810] INFO -- : worker=0 spawned pid=16810 I, [2017-02-13T00:58:12.461829 #16810] INFO -- : worker=0 ready

nginx.conf

user rails;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 1024;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        #  upstream unicorn {
        #    server unix:/home/rails/myapp/shared/sockets/unicorn.sock fail_timeout=0;
        #  }
        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

unicorn.rb

# set path to application
app_dir = File.expand_path("../..", __FILE__)
shared_dir = "#{app_dir}"
working_directory app_dir

print shared_dir

# Set unicorn options
worker_processes 2
preload_app true
timeout 30
print '1'

# Set up socket location
listen "#{shared_dir}/shared/sockets/unicorn.sock", :backlog => 64
print '2'

# Logging
stderr_path "#{shared_dir}/shared/log/unicorn.stderr.log"
stdout_path "#{shared_dir}/shared/log/unicorn.stdout.log"

print '3'
# Set master PID location
pid "#{shared_dir}/shared/pids/unicorn.pid"
print '4'

sites-available/default

upstream app {
    # Path to Unicorn SOCK file, as defined previously
    server unix:/home/rails/myapp/shared/sockets/unicorn.sock fail_timeout=0;
}

server {
    listen 80;
    server_name _;

    root /home/rails/myapp/public;

    try_files $uri/index.html $uri @app;

    location @app {
        proxy_pass http://app;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
    }

    error_page 500 502 503 504 /500.html;
    client_max_body_size 4G;
    keepalive_timeout 10;
}

Permissions on the socket:

srwxrwxrwx 1 rails rails 0 Feb 10 18:00 unicorn.sock

Any ideas on how I can track this down?

Answer 1

Your Rails application is failing to return a response to the Unicorn HTTP server within the amount of time configured by your timeout settings (currently 30 seconds, as configured by the timeout 30 line in your unicorn.rb).

Either increase the timeout setting in your unicorn.rb, or (preferably) debug why your Rails application is taking > 30 seconds to respond to a request.