Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Capistrano times out when deploying using Amazon RDS

I cannot seem to get Capistrano to play nicely with AmazonRDS. I've looked all over the place for any info on setting this up correctly, but haven't found any. Right now, when I cap deploy, the process times out.

This is my deploy.rb:

set :deploy_to, "/opt/bitnami/apps/annarbortshirtcompany.com/cms/" 
set :scm, :git
set :repository,  "ssh://[email protected]/~/repo/cms.git"
set :deploy_via, :remote_cache

set :user, "user"
ssh_options[:keys] = [File.join(ENV["HOME"], "EC2", "admin.pem")]
ssh_options[:forward_agent] = true
set :branch, "master"
set :use_sudo, true

set :location, "ec2-webserver.compute-1.amazonaws.com"
role :web, location
role :app, location
role :db, "cmsinstance.c7r8frl6npxn.us-east-1.rds.amazonaws.com", :primary => true

# If you are using Passenger mod_rails uncomment this:
namespace :deploy do
  task :start do ; end
  task :stop do ; end
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
  end
end

The username for the RDS database instance differs from the SSH username set here, but is defined in my database.yml. I figure that this is probably not being read by capistrano, but have no idea how to make that happen.

When I "cap deploy":

ubuntu@ubuntu-VirtualBox:~/RailsApps/cms$ cap deploy
  * executing `deploy'
  * executing `deploy:update'
 ** transaction: start
  * executing `deploy:update_code'
    updating the cached checkout on all servers
    executing locally: "git ls-remote ssh://[email protected]/~/repo/cms.git master"
    command finished in 1590ms
  * executing "if [ -d /app-directory/shared/cached-copy ]; then cd /app-directory/shared/cached-copy && git fetch -q origin && git fetch --tags -q origin && git reset -q --hard ffc4ec7762566f801c4a9140aa3980dc71e3d06f && git clean -q -d -x -f; else git clone -q  ssh://[email protected]/~/repo/cms.git /app-directory/shared/cached-copy && cd /app-directory/shared/cached-copy && git checkout -q -b deploy ffc4ec7762566f801c4a9140aa3980dc71e3d06f; fi"
    servers: ["ec2-webserver.compute-1.amazonaws.com", "dbinstance.us-east1.rds.amazonaws.com"]
*** [deploy:update_code] rolling back
  * executing "rm -rf /app-directory/releases/20110607161612; true"
    servers: ["ec2-webserver.compute-1.amazonaws.com", "dbinstance.us-east1.rds.amazonaws.com"]
 ** [deploy:update_code] exception while rolling back: Capistrano::ConnectionError, connection failed for: dbinstance.us-east1.rds.amazonaws.com (Errno::ETIMEDOUT: Connection timed out - connect(2))
    connection failed for: dbinstance.us-east1.rds.amazonaws.com (Errno::ETIMEDOUT: Connection timed out - connect(2))

Why would it want to "update the cached checkout on all servers"? The DB server shouldn't even be needed at this point. I am stumped at how to fix this. Hopefully someone can point me in the right direction!

like image 544
Tony Ghita Avatar asked Jun 07 '11 17:06

Tony Ghita


People also ask

What are some of the common causes why you Cannot connect to a DB instance on AWS?

When you can't connect to a DB instance, the following are common causes: Inbound rules – The access rules enforced by your local firewall and the IP addresses authorized to access your DB instance might not match. The problem is most likely the inbound rules in your security group.

How can I improve my RDS performance?

DB instance RAM recommendations An Amazon RDS performance best practice is to allocate enough RAM so that your working set resides almost completely in memory. The working set is the data and indexes that are frequently in use on your instance. The more you use the DB instance, the more the working set will grow.

What should you do to scale out in RDS database that has a read heavy workload?

You can easily create additional nodes from the Amazon RDS console or API. To scale your read operations, you horizontally scale your database through read replicas. When you create a read replica, Amazon RDS creates read-only copies of your database and manages the asynchronous replication from the primary database.

What is read latency in RDS?

Amazon RDS reports read and write latency separately at 1-minute intervals. Typical values for latency are in milliseconds (ms). Throughput – The number of bytes each second that are transferred to or from disk. This metric is reported as the average throughput for a given time interval.


1 Answers

I had this exactly problem and struggled with it for what I'm embarrassed to say was a good 5 or 6 hours. In the end, when I realized what the problem was I felt like smacking myself because I knew this once but had forgotten it. Here's the crux of the problem, starting with this part of deploy.rb:

set :location, "ec2-webserver.compute-1.amazonaws.com"
role :web, location
role :app, location
role :db, "cmsinstance.c7r8frl6npxn.us-east-1.rds.amazonaws.com", :primary => true

When you define the machine roles for Capistrano, you're not actually identifying which machines will play a particular role...rather, you're identifying on which machines the Capistrano code will run when applying a deployment recipe for a role. So, when you define the :db role, you want to point to your EC2 instance, not the RDS instance. You can't ssh into the RDS machine, so it's impossible for Capistrano to run a recipe there. Instead, point :db to the same machine as you're pointing :web and :app, i.e.

set :location, "ec2-webserver.compute-1.amazonaws.com"
role :web, location
role :app, location
role :db, location, :primary => true

How does the RDS machine then have any involvement? Well, it's the database.yml file that dictates which machine is actually running the RDBMS where the SQL needs to be executed. You just need to be sure you're setting the host: value for the target database, e.g.:

production:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: <your_db>_production
  pool: 5
  username: <username>
  password: <password>
  host: cmsinstance.c7r8frl6npxn.us-east-1.rds.amazonaws.com

Make sense?

I hope this save someone else the frustration I experienced.

  • David
like image 65
David L. Bean Avatar answered Sep 20 '22 13:09

David L. Bean