How to measure how much memory each gem requires at initialization?

Question

I have a Rails 2.3.10 app with bundler. At startup the memory footprint is quite big (300MB in development mode).

I would like to see how much memory each gem takes on startup.

Answer 1

We had a problem in which our basic Rails app, with no traffic or requests, had a footprint of ~140MB on startup.

We used the following approach to trace the memory requirement of each gem specified in the Gemfile of our app, without having to try to patch bundler.

1. using rails new myappname generate a new empty rails app
2. Copy the Gemfile from the main project to this new rails project
3. run bundle install and then rails server to ensure it is possible to boot up the rails server and that any basic configurations required are loaded
4. Open up the Gemfile and with the exception of the specification for the rails gem, append require: false at the end of each line. Ensure that any other gems that are specified with one name but required using :require => 'othergemname' are using the older ruby Hash notation so that the pattern match below will catch it.
5. Run bundle install again to regenerate the Gemfile.lock

6. Create the following script which will use manually require each gem specified in the Gemfile and log the system memory consumed by the rails process before and after.

   # require_and_profile.rb
   def require_and_profile(gemname = nil)
     unless gemname
       puts "%-20s: %10s | %10s" % ['gem','increment','total']
       return
     end
     # This is how to get memory of calling process in OS X, check host OS for variants
     memory_usage = `ps -o rss= -p #{Process.pid}`.to_i / 1024.0
     require gemname
     puts "%-20s: %10.2f | %10.2f" % [ gemname, (`ps -o rss= -p #{Process.pid}`.to_i / 1024.0 - memory_usage), (`ps -o rss= -p #{Process.pid}`.to_i / 1024.0)]
   end
   
   pattern = /^[^#]*gem[ ]*['"]([^,'"]*)['"][ ,~>0-9\.'"]*(:require[ => ]*['"]([^'"]*)['"][, ])?/
   
   require_and_profile
   File.open('Gemfile').each do |line|
     if line.match(pattern)
     if line.match(pattern)[3]
       require_and_profile line.match(pattern)[3]
     else
         require_and_profile line.match(pattern)[1]
       end
     end
   end
   

7. Run rails c

8. load 'require_and_profile.rb'
9. The output shows how much (in MB) each gem adds to the base app footprint (increment) and what the total footprint is after inclusion of the gem (total).

This helped us identify for example, that we'd been requiring asset-sync in our boot when we only needed it in the :asset group. We do find that on different boot-ups the memory footprint of each gem is not exactly the same, but running it a few times does show you the patterns of which ones are the memory-hungry gems.

Answer 2

There is an easier way to do this now with the derailed gem:

add to your gemfile:

gem 'derailed', group: :development

then on the command line from your apps root:

bundle exec derailed bundle:mem

This will print out how much memory each gem takes as it's included.