Rails stops responding if Idle

Question

I've been having this problem for a while and searched a lot about this, but nothing seems to solve my problem.

I have an application running in Ruby on Rails with NGINX and UNICORN everything with Docker.

If I run the application (docker-compose up) the web site runs perfectly for a few minutes. If the web site stays idle for about 5 to 10 minutes, and I send another request the rails application stops responding giving time out from Unicorn.

The weird part is that every application I develop, have the same problem and the only thing in common these applications have are the Gemfile.

What I found so far is that when the request is sent, NGINX receives this request, pass it to Unicorn, Unicorn receives it and passes to ruby... which doesn't answer and then Unicorn gives me the 502 bad request.

I'm really lost at this one.

this is my NGINX config file:

upstream applicationName {
    # Path to Puma SOCK file, as defined previously
    server unix:/tmp/applicationName.sock fail_timeout=0;
}

server {
  listen 80;
  server_name dev.applicationName.com.br;

  location / {
    autoindex on;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    # time out settings
    proxy_connect_timeout 159s;
    proxy_send_timeout   600;
    proxy_read_timeout   600;
    proxy_buffer_size    64k;
    proxy_buffers     16 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass_header Set-Cookie;
    proxy_redirect     off;
    proxy_hide_header  Vary;
    proxy_set_header   Accept-Encoding '';
    proxy_ignore_headers Cache-Control Expires;
    proxy_set_header   Referer $http_referer;
    proxy_set_header   Host   $host;
    proxy_set_header   Cookie $http_cookie;
    proxy_set_header   X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://applicationName;
  }
}

And this is my unicorn config

@dir = File.expand_path(File.dirname(__FILE__)) + "/.."

worker_processes 2
working_directory @dir

timeout 10

listen File.join('/tmp/applicationName.sock')

preload_app true# if ENV['RAILS_ENV'] != 'development'

GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

check_client_connection false

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

This is the Gemfile (nothing wrong with it)

source 'https://rubygems.org'
gem 'rails', '4.2.4'
gem 'unicorn-rails', '~> 2.2'
gem 'pg'
gem 'mysql2', '~> 0.3.18'
gem 'sass-rails', '~> 5.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.1.0'
gem 'duktape'
gem 'jquery-rails'
gem 'turbolinks'
gem 'jbuilder', '~> 2.0'
gem 'bootstrap-sass'
gem 'devise'
gem 'simple_form'
gem 'minitest'
gem "paperclip", "~> 4.3"
gem 'aws-sdk', '< 2.0'
gem 'mail_form', '~> 1.5.0.rc'
gem 'sendgrid-ruby'
gem 'zopim_rails'
gem 'meta-tags'
gem 'ckeditor'
gem 'slick_rails'

group :development do
  gem 'better_errors'
  gem 'binding_of_caller', :platforms=>[:mri_20]
  gem 'quiet_assets'
  gem 'rails_layout'
  gem 'spring-commands-rspec'
  gem 'web-console', '~> 2.0'
  gem 'spring'
end
group :production do
  gem 'therubyracer'
end
group :development, :test do
  gem 'factory_girl_rails'
  gem 'faker'
  gem 'pry-rails'
  gem 'pry-rescue'
  gem 'rspec-rails'
  gem 'rubocop'
  gem 'byebug'
end

group :test do
  gem 'capybara'
  gem 'database_cleaner'
  gem 'launchy'
  gem 'selenium-webdriver'
end

The log on the server shows me this, when the error happens:

nginx_1 | 172.17.0.1 - - [05/Feb/2016:12:52:41 +0000] "GET /products HTTP/1.1" 502 574 "http://dev.nutrimais.com.br/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36"
web_1   | E, [2016-02-05T12:52:41.551728 #1] ERROR -- : reaped #<Process::Status: pid 9 SIGKILL (signal 9)> worker=0

And on the development log:

Started GET "/products" for 127.0.0.1 at 2016-02-05 12:52:18 +0000
Cannot render console from 127.0.0.1! Allowed networks: 127.0.0.1, ::1, 127.0.0.0/127.255.255.255

Answer 1

Your fix of moving the MySQL to the same server also fixed the issue for me but that's not suitable for my production environment so I dug deeper into what made the difference with that setup.

Turned out it was nothing related to the application stack or configuration, but a NAT timeout that was killing my sessions.

I had my server on Azure and my database in AWS. Azure has a 4 minute unconfigurable NAT timeout on outbound connections. That means if a database connection was idle for more than 4 minutes, Azure would just silently kill the port mapping and any traffic the Rails app tried to send down that port silently went into a black hole.

The fix was per this blob at the OS level on the Rails server: Drop the TCP keepalive interval to half the NAT timeout, and adjust the keepalive interval and retransmissions to appropriate values for the new keepalive interval.

For Linux, you should change these kernel variables as follows:

sudo sysctl net.ipv4.tcp_keepalive_time = 120     
sudo sysctl net.ipv4.tcp_keepalive_intvl = 30 
sudo sysctl net.ipv4.tcp_keepalive_probes = 8

For Windows, you should change the registry values below.

All DWORDs in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters:

KeepAliveInterval = 30 
KeepAliveTime = 120 
TcpMaxDataRetransmissions = 8