Server hangs all requests after a while

Question

Our Rails 4.0 application (Ruby 2.1.2) is running on Nginx with Puma 2.9.0.

I recently noticed that all requests to our application hang after a while (usually 1 or 2 days).

When checking the log, which is set to debug mode, I noticed the following log stack up:

[2014-10-11T00:02:31.727382 #23458]  INFO -- : Started GET "/" for ...

It does mean that requests actually hit the Rails app but somehow it isn't proceeded, while normally it would be:

I, [2014-10-11T00:02:31.727382 #23458]  INFO -- : Started GET "/" for ....
I, [2014-10-11T00:02:31.729393 #23458]  INFO -- : Processing by HomeController#index as HTML

My puma config is the following:

threads 16,32
workers 4

Our application is only for internal usage as now, so the RPM is very low, and none of the requests are take longer than 2s.

What is/are the reasons that could lead to this problem? (puma config, database connection, etc.)

Thank you in advance.

Update: After installing the gem rack_timer to log the time spent on each middleware, I realized that our requests has been stuck at the ActiveRecord::QueryCache when the hang occurred, with huge amount of time on it:

Rack Timer (incoming) -- ActiveRecord::QueryCache: 925626.7731189728 ms

I removed this middleware for now and it seems to be back to normal. However, I understand the purpose of this middleware is to increase the performance, so removing it is just a temporary solution. Please help me find out the possible cause of this issue.

FYI, we're using mysql (5.1.67) with adapter mysql2 (0.3.13)

Answer 1

It could be a symptom of RAM starvation due to the query cache getting too big. We saw this in one of our apps running on Heroku. The default query cache is set to 1000. Lowering the limit eased the RAM usage for us with no noticeable performance degradation:

database.yml:

default: &default
  adapter: postgresql
  pool: <%= ENV["DB_POOL"] || ENV['MAX_THREADS'] || 5 %>
  timeout: 5000
  port: 5432
  host: localhost
  statement_limit: <%= ENV["DB_STATEMENT_LIMIT"] || 200 %>

However searching for "activerecord querycache slow" returns other causes, such as perhaps outdated versions of Ruby or Puma or rack-timeout: https://stackoverflow.com/a/44158724/126636

Or maybe a too large value for read_timeout: https://stackoverflow.com/a/30526430/126636