Rails 4 : ConnectionTimeoutError on heroku basic plan

Question

We are working on a simple Rails 4 webapp, hosted by Heroku with the free plan.

Recently, we are experiencing some ActiveRecord::ConnectionTimeoutError - one every few days - and we don't really understand the reasons behind it.

You may want to have a look at the full stack here.

We know Heroku sets a pool limit of 20 connections to the Postgres db, and this is most likely the the reason why the exception is thrown. This was our guess:

We are using Herku scheduler to launch a rake task every hour and, since that task queries the database, the problem could lie in the fact that those connections are not released from the pool after being used. So we tried to put everything inside a block:

ActiveRecord::Base.connection_pool.with_connection do

  #code with queries

end

But this doesn't seem to solve the problem. Do you have better suggestions / explanations?

Any help and/or insight is greatly appreciated, also on how we should debug this problem.

---

PS: We also tried to add the following code in a dedicated initializer, but it didn't solve the problem

Rails.application.config.after_initialize do
  ActiveRecord::Base.connection_pool.disconnect!

  ActiveSupport.on_load(:active_record) do
    config = ActiveRecord::Base.configurations[Rails.env] ||
                Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    config['pool']              = ENV['DB_POOL']      || 20
    ActiveRecord::Base.establish_connection(config)
  end
end

Answer 1

In my experience these issues will come up on basic Heroku plans. There are a number of ways to mitigate the frequency without added cost. But you will never get to 0% occurrance with the basic/hobby stack.

Here's why in my experience.. until you hit 'premium-yanari' you wont get High Availability. You can get 99.95% expected uptime vs the free 99.5%. Doens't sound like much, but Heroku charges $150 extra for a reason (compared to the 'standard-yanari' plan).

After all the optimizations with connection pooling, reaping frequency, and forking, if your process cannot reach the database server you'll get a timeout. If this is the only cause, it will happen very infrequently, which may be your case. Only until I upgraded to 'premium-yanari' did this seemingly random timeout stop occurring.

I'm not suggestion you start paying $200/mo as a solution. You can get to near zero with a few suggestions below. Tuning concurrency, db connection, and experimenting with various options like timeouts take some trail and error - as there seems to be no plug-and-chug formula to give you a perfect combination for all these parameters.

1) Move to Unicorn to take more control of your processes and database connections. Here is a sample unicorn.rb file https://gist.github.com/blotto/8005531b9c94f3732240

2) Increase your timeouts ( currently 5 seconds ). See unicorn.rb :

 timeout ENV['REQUEST_TIMEOUT'].to_i > 0 ?  ENV['REQUEST_TIMEOUT'].to_i : 15

3) Be sure you're on the latest stable Rails 4.1.1. As 4.0.x did have some Postgres adaptor issues. See https://github.com/rails/rails/issues/12867