Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to troubleshoot memory bloat at boot for rails app?

I am having memory issues with a Rails app on Heroku running ruby 2.3.3 and Rails 4.1.13. When I load the app on Heroku it starts at around 430 Mb but is quite stable there - but still reached 512 Mb on Heroku (as would be expected). My main question is: How do I pinpoint and reduce memory usage during BOOT on my app? I.e. how do I reduce the amount of memory used at startup?

Heroku memory usage

I have looked into most gems, e.g. derailed, oink, memory profiler, and stackprof but what has helped me the most has been Scout add-on on Heroku in terms of finding memory bloats and memory hungry controllers etc. Through this I have managed to reduce the amount the application grows in terms of memory but not so much how much memory it uses at startup.

I know gems are loaded on startup and the more, and larger, the gems are the more memory is used at startup. Running bundle exec derailed bundle:mem gives me this:

TOP: 92.3008 MiB
  carrierwave-aws: 16.7461 MiB
    carrierwave/storage/aws: 16.6094 MiB
      aws-sdk-core/s3: 7.9414 MiB
      aws-sdk-resources: 2.5703 MiB
        aws-sdk-core: 2.5117 MiB
          jmespath: 1.8867 MiB
            jmespath/nodes: 1.1406 MiB
              jmespath/nodes/function: 0.3867 MiB
      seahorse/client/base: 0.9414 MiB
        seahorse/client/plugins/net_http: 0.5273 MiB
      aws-sdk-core/xml/parser/engines/rexml: 0.8633 MiB
        rexml/document: 0.8516 MiB
          rexml/element: 0.5508 MiB
  rails/all: 13.6992 MiB
    rails: 8.9453 MiB (Also required by: active_record/railtie, active_model/railtie, and 6 others)
      rails/application: 7.2188 MiB
        rails/engine: 6.8672 MiB (Also required by: coffee/rails/engine)
          rails/railtie: 6.5898 MiB (Also required by: sprockets/railtie, jbuilder/railtie)
            rails/configuration: 6.3398 MiB (Also required by: rails/railtie/configuration)
              active_support/core_ext/object: 6.0586 MiB
                active_support/core_ext/object/conversions: 4.7227 MiB
                  active_support/core_ext/hash/conversions: 4.0039 MiB (Also required by: active_support/core_ext/hash, active_record/serializers/xml_serializer)
                    active_support/time: 3.8008 MiB (Also required by: active_record/base)
                      active_support/core_ext/time: 3.0938 MiB
                        active_support/core_ext/time/calculations: 2.9922 MiB (Also required by: active_support/core_ext/numeric/time, active_support/core_ext/string/conversions)
                          active_support/core_ext/time/conversions: 2.2148 MiB (Also required by: active_support/core_ext/time, active_support/core_ext/date_time/conversions)
                            active_support/values/time_zone: 2.1602 MiB (Also required by: active_support/time_with_zone, active_support/core_ext/date_time/conversions)
                              tzinfo: 1.8984 MiB
                  active_support/core_ext/array/conversions: 0.6211 MiB (Also required by: active_support/duration, action_dispatch/http/request, and 2 others)
                    active_support/xml_mini: 0.5508 MiB (Also required by: active_support/core_ext/hash/conversions)
                active_support/core_ext/object/json: 1.0352 MiB (Also required by: active_support/json/encoding)
                  json: 0.7227 MiB (Also required by: active_support/json/decoding, pg/text_encoder, and 19 others)
                    json/common: 0.3906 MiB (Also required by: json/ext)
      active_support: 0.7109 MiB (Also required by: active_support/time, active_support/railtie, and 5 others)
        active_support/dependencies/autoload: 0.332 MiB (Also required by: rails, active_support/rails)
          active_support/inflector/methods: 0.332 MiB (Also required by: active_support/core_ext/string/inflections, active_support/core_ext/time/conversions, and 5 others)
            active_support/inflections: 0.3164 MiB (Also required by: active_support/inflector)
        active_support/logger: 0.3281 MiB
      action_dispatch/railtie: 0.668 MiB (Also required by: action_controller/railtie)
        action_dispatch: 0.6289 MiB (Also required by: action_controller)
          active_support/rails: 0.4414 MiB (Also required by: active_record, active_model, and 4 others)
            active_support/deprecation: 0.418 MiB (Also required by: sass/rails/helpers, formtastic/deprecation)
    active_record/railtie: 2.8555 MiB
      active_record: 1.9766 MiB (Also required by: friendly_id)
        active_record/connection_adapters/abstract_adapter: 0.8516 MiB
        arel: 0.8086 MiB (Also required by: active_record/base)
          arel/visitors: 0.3438 MiB
      action_controller/railtie: 0.875 MiB (Also required by: rails/all, sprockets/railtie)
        action_controller: 0.6797 MiB (Also required by: heroku_rails_deflate/serve_zipped_assets)
          action_controller/metal/live: 0.332 MiB
    sprockets/railtie: 1.8516 MiB (Also required by: sass/rails/railtie)
      sprockets/rails/helper: 0.918 MiB
        action_view/helpers: 0.8945 MiB (Also required by: action_view/base)
          action_view/helpers/form_helper: 0.3828 MiB (Also required by: action_view/helpers/form_options_helper)
      sprockets: 0.9102 MiB (Also required by: sprockets/rails/helper)
        sprockets/directive_processor: 0.5586 MiB
          yaml: 0.5078 MiB (Also required by: active_support/ordered_hash, hirb, and 7 others)
            psych: 0.5078 MiB
  rails3-jquery-autocomplete: 9.9102 MiB
    action_controller/base: 8.4922 MiB
      action_controller/metal: 1.8281 MiB
        abstract_controller/base: 1.0898 MiB
          erubis: 0.8477 MiB (Also required by: action_view/template/handlers/erb, heroics)
            erubis/engine: 0.4063 MiB (Also required by: erubis/engine/eruby)
        action_dispatch/middleware/stack: 0.6367 MiB
          active_support/dependencies: 0.5703 MiB (Also required by: abstract_controller/helpers, active_record/base)
      abstract_controller/rendering: 1.5625 MiB
        action_view/view_paths: 1.4531 MiB (Also required by: action_view/rendering)
          action_view/base: 1.3477 MiB
            action_view/lookup_context: 0.5156 MiB
            action_view/template: 0.3789 MiB (Also required by: action_view/template/resolver)
      action_controller/metal/redirecting: 1.0586 MiB
        action_controller/metal/rack_delegation: 0.9297 MiB
          action_dispatch/http/request: 0.8555 MiB
      action_controller/metal/request_forgery_protection: 0.5859 MiB
      action_controller/metal/params_wrapper: 0.5 MiB
        action_dispatch/http/mime_type: 0.375 MiB (Also required by: abstract_controller/collector, jbuilder/jbuilder_template)
      action_controller/metal/url_for: 0.3008 MiB
    rails3-jquery-autocomplete/formtastic: 1.2578 MiB
      formtastic/inputs/base: 0.8125 MiB
  fog: 8.6523 MiB
    fog/vcloud_director: 1.0156 MiB
      fog/vcloud_director/compute: 0.9961 MiB
        fog/vcloud_director/query: 0.4453 MiB
          pp: 0.418 MiB (Also required by: scout_apm)
    fog/openstack: 0.8398 MiB
    fog/cloudstack: 0.5859 MiB
      fog/cloudstack/compute: 0.5859 MiB
    fog/internet_archive: 0.543 MiB
      fog/internet_archive/storage: 0.5352 MiB
        fog/internet_archive/core: 0.3125 MiB
    fog/rackspace: 0.3867 MiB
    fog/cloudsigma: 0.3594 MiB
      fog/cloudsigma/compute: 0.3047 MiB
    fog/vcloud: 0.3203 MiB
      fog/vcloud/compute: 0.3125 MiB
    fog/aws: 0.3125 MiB
  twitter: 8.3711 MiB
    addressable/uri: 4.2422 MiB (Also required by: twitter/base, twitter/rest/request, and 8 others)
      addressable/idna: 2.7891 MiB
        addressable/idna/pure: 2.7461 MiB
      public_suffix: 0.3242 MiB
    twitter/streaming/client: 1.7266 MiB
      twitter/streaming/response: 1.4453 MiB
        http: 1.418 MiB
          http/response: 1.2109 MiB
            http/cookie_jar: 0.9219 MiB
              http/cookie: 0.9102 MiB
                domain_name: 0.8203 MiB
                  domain_name/etld_data: 0.7813 MiB
    twitter/configuration: 0.8477 MiB (Also required by: twitter/rest/help)
      twitter/base: 0.6016 MiB (Also required by: twitter/entity, twitter/identity, and 12 others)
        twitter/null_object: 0.4688 MiB (Also required by: twitter/trend_results)
          naught: 0.4063 MiB
    twitter/cursor: 0.8164 MiB (Also required by: twitter/rest/utils, twitter/rest/friends_and_followers, and 2 others)
      twitter/rest/request: 0.7852 MiB (Also required by: twitter/rest/utils, twitter/rest/friends_and_followers, and 8 others)
        faraday: 0.6406 MiB (Also required by: twitter/rest/client, twitter/rest/response/parse_json, and 6 others)
    twitter/rest/client: 0.418 MiB
      twitter/rest/api: 0.3828 MiB
  asset_sync: 4.9609 MiB
    asset_sync/multi_mime: 4.5039 MiB
      mime/types: 4.4883 MiB (Also required by: mime/types/columnar)
        mime/types/registry: 4.1914 MiB
    asset_sync/storage: 0.4375 MiB
      fog/core: 0.4063 MiB (Also required by: fog, fog/xml, and 48 others)
  platform-api: 4.5273 MiB
    platform-api/client: 2.8359 MiB
  newrelic_rpm: 3.875 MiB
    new_relic/control: 3.875 MiB
      new_relic/agent: 3.4883 MiB
        new_relic/agent/agent: 1.7422 MiB
          new_relic/agent/configuration/manager: 0.4961 MiB (Also required by: new_relic/agent/configuration)
            new_relic/agent/configuration/default_source: 0.3633 MiB
        new_relic/agent/transaction_sampler: 0.3516 MiB
  carrierwave: 2.8281 MiB (Also required by: carrierwave-aws)
    carrierwave/uploader: 1.7656 MiB
  friendly_id: 2.5273 MiB
    friendly_id/object_utils: 2.4453 MiB
      active_record/base: 2.4258 MiB
        active_record/querying: 0.3906 MiB
  compass: 2.1328 MiB (Also required by: compass-rails)
    compass/sass_extensions: 1.2461 MiB
      compass/sass_extensions/sprites: 0.6875 MiB
        compass/sass_extensions/sprites/engines: 0.4609 MiB
          compass/sass_extensions/sprites/engines/chunky_png_engine: 0.4492 MiB
            chunky_png: 0.4492 MiB
      compass/sass_extensions/functions: 0.5078 MiB
    compass/configuration: 0.6172 MiB
      compass/configuration/data: 0.6133 MiB
  pg: 2.1055 MiB
    pg_ext: 1.9102 MiB
  feedjira: 2.0781 MiB
    loofah: 0.625 MiB
    sax-machine: 0.4414 MiB
  sass-rails: 1.8047 MiB
    sass/rails: 1.8008 MiB
      sass/rails/helpers: 1.7578 MiB
        sprockets/sass_functions: 1.7383 MiB
          sass: 1.7383 MiB (Also required by: sass/rails/importer, sprockets/sass_importer)
            sass/engine: 1.4492 MiB
              sass/script: 0.5586 MiB (Also required by: sass/script/css_parser)
              sass/scss: 0.3281 MiB
                sass/scss/parser: 0.3047 MiB
  scout_apm: 1.418 MiB
    webrick: 0.5664 MiB
  jbuilder: 1.1016 MiB (Also required by: jbuilder/jbuilder_template)
    active_support/cache: 0.3906 MiB
  rmagick: 1.0234 MiB
    rmagick_internal.rb: 1.0195 MiB
      RMagick2.so: 0.6875 MiB
  lazyload-rails: 0.8555 MiB
    nokogiri: 0.8555 MiB (Also required by: TOP, sax-machine/handlers/sax_nokogiri_handler, and 5 others)
  sitemap_generator: 0.5195 MiB
  dalli: 0.4648 MiB
  unicorn: 0.457 MiB
  hirb: 0.3984 MiB

I imagine 92 Mb is quite a bit in terms of gems but I do use them all in the app. Carrierwave is the largest, but very important in how I cache images on AWS (as well as with Fog) and so on. This only accounts for about 1/4 of the memory used at boot though.

In my application controller I have a before filter where I load some variables that are used throughout the website (in global headers, sidebars etc) like this:

@search = TextSearch.new   
@administrator_logged_in = current_administrator    
@nbr_of_stores_total = Store.active.size # Used in header
@nbr_of_blog_posts_total = BlogPost.once_posted.size # Used in header
contests_open = Contest.active.open.size
@active_contests_string = contests_open > 0 ? " (" << contests_open.to_s << ")" : ""
add_breadcrumb "Home", root_path
@open_graph_image = "https://s3.eu-central-1.amazonaws.com/myapp/assets/myapp_logo_large.png"

# Season-related
@is_christmas = false
@gift_word_seasonal = "christmas presents"

I would imagine this would be loaded into boot (and overall) memory, right?

Where else, and how, should I look for startup memory-usage? In short:

  1. Where in my rails app should I trouble boot memory (like middleware, initializers etc)?

  2. What tools (such as the derailed command above) can I use to find/pinpoint memory hungry calls/actions?

  3. I am having trouble finding the direct cause and effect on my trouble shooting actions. Like, finding the answer to "if I remove this code, how much memory do I save?". How should I measure/benchmark any actions I do on my app (rather than upload and check the overall usage in the graph)?

like image 641
Christoffer Avatar asked Jan 08 '18 08:01

Christoffer


1 Answers

Try running GC.stat command This would give info on memory usage and garbage collection in the app based on which you can decide.

Do memory profiling https://github.com/ice799/memprof could also be of help to you

like image 192
Kalaivasakan Avatar answered Oct 13 '22 01:10

Kalaivasakan