Elastic Beanstalk Ruby/Rails need to install git so bundle install works.. but is not

Question

I'm having an issue deploying our rails app.. I created a hook like the example on the AWS blog howto http://ruby.awsblog.com/post/Tx2AK2MFX0QHRIO/Deploying-Ruby-Applications-to-AWS-Elastic-Beanstalk-with-Git like:

packages:
  yum:
    git: []

even I run a bundle package to create vendor/cache to have all the gems there... and still getting: git://github.com/refinery/refinerycms-search.git (at 2-0-stable) is not checked out. Please run bundle install (Bundler::GitError)

any help will be nice, we trying to move all our apps to EB. but seens that git does not install or something is going on.. I need git on the EB ec2 instance it creates.

StackTrace:

Error message:
git://github.com/refinery/refinerycms-search.git (at 2-0-stable) is not checked out.         Please run `bundle install` (Bundler::GitError)
Exception class:
PhusionPassenger::UnknownError
Application root:
/var/app/current
Backtrace:
#   File    Line    Location
0   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/source.rb 801 in        `rescue in load_spec_files'
1   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/source.rb 799 in   `load_spec_files'
2   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/source.rb 381 in `local_specs'
3   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/source.rb 774 in `specs'
4   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/definition.rb 174 in `block in resolve'
5   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/definition.rb 172 in `each'
6   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/definition.rb 172 in `resolve' 
7   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/definition.rb 113 in `specs'
8   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/definition.rb 158 in `specs_for'
9   /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/definition.rb 147 in `requested_specs'
10  /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/environment.rb    23  in `requested_specs'
11  /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb    11   in `setup'
12  /usr/share/ruby/1.9/gems/1.9.1/gems/bundler-1.2.1/lib/bundler.rb    116 in `setup'
13  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/utils.rb    326 in `prepare_app_process'
14  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/rack/application_spawner.rb 156 in `block in initialize_server'
15  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/utils.rb    563 in `report_app_init_status'
16  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/rack/application_spawner.rb 154 in `initialize_server'
17  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-  1002/support/lib/phusion_passenger/abstract_server.rb    204 in `start_synchronously'
18  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/abstract_server.rb  180 in `start'
19  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/rack/application_spawner.rb 129 in `start'
20  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/spawn_manager_orig.rb   253 in `block (2 levels) in spawn_rack_application'
21  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/abstract_server_collection.rb   132 in `lookup_or_add'
22  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/spawn_manager_orig.rb   246 in `block in spawn_rack_application'
23  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/abstract_server_collection.rb   82  in `block in synchronize'
24      prelude>    10:in `synchronize'
25  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/abstract_server_collection.rb   79  in `synchronize'
26  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/spawn_manager_orig.rb   244 in `spawn_rack_application'
27  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/spawn_manager_orig.rb   137 in `spawn_application'
28  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/spawn_manager.rb    16  in `spawn_application_with_env'
29  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/spawn_manager_orig.rb   275 in `handle_spawn_application'
30  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/abstract_server.rb  357 in `server_main_loop'
31  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/lib/phusion_passenger/abstract_server.rb  206 in `start_synchronously'
32  /var/lib/passenger-standalone/3.0.17-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/support/helper-scripts/passenger-spawn-server

UPDATE more info: I decided to log into the instance with ec2-user and notice that git is installed the gems I need ARE installed so far I can see.. when I do bundle list they all show up even the one that the error is complaining about.. I do notice that when I do a general gem list then it DOES not show up... but it should be using the bundle one.

Answer 1

(Note that the following workaround should only be used if you must use Git sources for dependencies. It is recommended not to install dependencies from external Git repositories if it can be avoided. See below for details on why that is.)

When using Git backed libraries in a Gemfile with Passenger, you must disable shared gems in an installation (in addition to installing Git in the hook you listed above). You can do this by setting the BUNDLE_DISABLE_SHARED_GEMS Bundler environment variable in your existing .ebextensions/ruby.config file like so:

option_settings:
  - option_name: BUNDLE_DISABLE_SHARED_GEMS
    value: "1"
  - option_name: BUNDLE_PATH
    value: "vendor/bundle"

packages:
  yum:
    git: []

Disabling shared gems will force all dependencies to be vendored into your application in vendor/bundle as specified by the BUNDLE_PATH variable.

Note that, whenever possible, you should avoid installing public libraries from Git sources with your application. Using Git for library locations introduces another point of failure for a deployment install, since the Git repository may be temporarily unavailable or even permanently moved. Also keep in mind that forcing vendored installs in a deployment will cause your Elastic Beanstalk deployments to be much slower on subsequent deploys of an app with the same dependencies. This is because the libraries will be re-installed at each deploy instead of taking advantage of the system-wide installation that Elastic Beanstalk has Bundler perform by default.

In short, if there is an official RubyGem release of the library in question, you should use that version instead; and if not, you should suggest to the library author that an official RubyGem release be made available.

FYI a similar question about this Git problem with regular Passenger/Rails deployments was previously asked: Rails 3: Passenger can't find git gems installed by bundler

Answer 2

Another option is to package the gem source directly with your application and then point bundler at that.

Copy the gem source into vendor/gems/mygem

Then, in your Gemfile:

gem 'mygem', path: File.join(File.dirname(__FILE__), 'vendor', 'gems', 'mygem')

See more here: http://viget.com/extend/bundler-best-practices

Answer 3

Amazon's Elastic Beanstalk Ruby AMI needs a little tweaking in order to allow you to bundle gems from git without sacrificing deployment speed, behavior you get out of the box with Capistrano and Heroku.

Fortunately, the Elastic Beanstalk configuration API makes the necessary tweaks possible without requiring you to maintain a custom AMI.

Here's the Elastic Beanstalk configuration that I use to get the desired, conventional Ruby deployment behavior with Amazon's own AMI: https://github.com/gkop/elastic-beanstalk-ruby .

Answer 4

After trying the accepted answer, I found that a simpler .ebextensions/ruby.config was the only config that worked:

packages:
  yum:
    git: []

Answer 5

• ok well after lot of research and testing I figure this was related to Amazon having some issues with the passenger env values...

• I was able to run manually rails s and then it worked fine and all gems loaded.. but with passenger it did not so find that if I run

  bundle pack --all

• --all so it also packs the git gems..

• it will then run like a charm..

• to be able to have this run in the mean time Amazon fixes this issue with bundle I create a hook and force it to run after every install. not the best way but works.

Note: using the hook I cant seen to run the command on EB updates so I git add the vendor/cache and it uploads it all by default.