Starting Ruby app terribly slow due to requires from GEM_HOME

Question

I am currently working on a ruby application, but it is running very (Very!) slow.. Until now, I've tried a couple of things and I could narrow it down to the main issue: Ruby is trying to look for it's requires in every single directory in the $LOAD_PATH.

Basically what I am observing, is that ruby is looking through a lot of files, trying to see if it's requires exist over there. In case it wont find them, it will go to the next directory in line. The nice thing is I can see this happening with strace. There is a lot of output like this:

open("/boa_proj_build/nsteen/.gem/gems/i18n-0.7.0/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/thread_safe-0.3.5/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/tzinfo-1.2.2/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/minitest-5.8.2/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/activesupport-4.2.4/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/climate_control-0.0.3/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/cocaine-0.5.7/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/boa_loggable-0.2.2/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/ruby_expect-1.6.0/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/cctools-3.0.1/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/git-1.2.9.1/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/naught-1.1.0/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/symbolizer-0.0.1/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/settingslogic-2.0.9/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/memoist-0.12.0/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/highline-1.7.8/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/boa_proj_build/nsteen/.gem/gems/commander-4.3.5/lib/commander/help_formatters/base.rb", O_RDONLY|O_CLOEXEC) = 8

As you can see, it is looking through some directories to find it's requirements.

Tracing this with a test application, filtering for the ENOENT error and counting the occurances, displays the following concerning output:

vdi9442:/boa_proj_build/nsteen/$ strace packager --version 2>&1 | grep ENOENT | wc -l
3454261

of-course, 3.5 million is a LOT. And this will result in a load time of about 5 minutes (and about half without the strace), before it will output it's version number (default functionality from the commander gem).

I've removed my entire gem home directory, and ran the test again, and it is immediately faster, but I can see it looking through those few gems (dependencies like commander) again, but 'just' a several thousand occurances instead of 3.5 mil.

My gem env looks like this:

  - GEM PATHS:
     - /boa_proj_build/nsteen/.gem
     - /home/nsteen/.gem/ruby/2.1.0
     - /cadappl/ruby/2.1.1/ruby/lib/ruby/gems/2.1.0

It looks like ruby is just walking through my entire load path, to satisfy some dependencies. It's fine, but this is just getting rediculous. Does anybody have a clue what's going on? This can't be wanted/default behaviour I suspect?

Does anybody have a clue what's going on? And how I can speed things up?

Answer 1

There are quite a few ways to deal with this including gemrc files that I won't get into. I will mention a few of the options you have:

Solution 1:

The other answer is correct, but I wanted to expand a bit on this topic since it is one that people seem to often face. RVM can help. One particular feature is made for this, gemsets. I personally have moved to rbenv and have not looked back. Rbenv is a lot less intrusive on your environment than RVM, but they are both great. You can use gemsets in both, RVM and rbenv, to limit what gems are available to your app. You can make an app specific gemset too. With a gemset, your app will look in one location and will only load gems that it has used. It won't mix with other apps gemsets. The best thing about it is automatic, you just need to set it up once and it will auto switch when you are in that directory. It is also integrated with Rubymine and other IDEs/Editors that support Ruby.

Solution 2:

Set GEM_PATH environment variable to one of the locations. That overrides everything else so it at least will stop looking in multiple places.

Solution 3(Application loading optimization):

Something you might look into is if you are using bundler, you may want to only require special use gems when they are needed. For example, if you are using the curb gem in one module of your app, add this to top of the file loading your module:

require 'curb'

and change your Gemfile so the line loading curb looks like this:

gem 'curb', require: false

The less gems you load when your app starts the faster it will load. The auto require feature is great, but is often forgotten and taken for granted. It is amazing the difference it could make.

Answer 2

I would suggest using rvm for ruby version and gem management. With rvm you can create application specific gemsets and ruby versions.

I hope this will solve your problem.

https://rvm.io/