Menu
I am having memory issues with a Rails app on Heroku running ruby 2.3.3 and Rails 4.1.13. When I load the app on Heroku it starts at around 430 Mb but is quite stable there - but still reached 512 Mb on Heroku (as would be expected). My main question is: How do I pinpoint and reduce memory usage during BOOT on my app? I.e. how do I reduce the amount of memory used at startup?
I have looked into most gems, e.g. derailed, oink, memory profiler, and stackprof but what has helped me the most has been Scout add-on on Heroku in terms of finding memory bloats and memory hungry controllers etc. Through this I have managed to reduce the amount the application grows in terms of memory but not so much how much memory it uses at startup.
I know gems are loaded on startup and the more, and larger, the gems are the more memory is used at startup. Running bundle exec derailed bundle:mem gives me this:
TOP: 92.3008 MiB
carrierwave-aws: 16.7461 MiB
carrierwave/storage/aws: 16.6094 MiB
aws-sdk-core/s3: 7.9414 MiB
aws-sdk-resources: 2.5703 MiB
aws-sdk-core: 2.5117 MiB
jmespath: 1.8867 MiB
jmespath/nodes: 1.1406 MiB
jmespath/nodes/function: 0.3867 MiB
seahorse/client/base: 0.9414 MiB
seahorse/client/plugins/net_http: 0.5273 MiB
aws-sdk-core/xml/parser/engines/rexml: 0.8633 MiB
rexml/document: 0.8516 MiB
rexml/element: 0.5508 MiB
rails/all: 13.6992 MiB
rails: 8.9453 MiB (Also required by: active_record/railtie, active_model/railtie, and 6 others)
rails/application: 7.2188 MiB
rails/engine: 6.8672 MiB (Also required by: coffee/rails/engine)
rails/railtie: 6.5898 MiB (Also required by: sprockets/railtie, jbuilder/railtie)
rails/configuration: 6.3398 MiB (Also required by: rails/railtie/configuration)
active_support/core_ext/object: 6.0586 MiB
active_support/core_ext/object/conversions: 4.7227 MiB
active_support/core_ext/hash/conversions: 4.0039 MiB (Also required by: active_support/core_ext/hash, active_record/serializers/xml_serializer)
active_support/time: 3.8008 MiB (Also required by: active_record/base)
active_support/core_ext/time: 3.0938 MiB
active_support/core_ext/time/calculations: 2.9922 MiB (Also required by: active_support/core_ext/numeric/time, active_support/core_ext/string/conversions)
active_support/core_ext/time/conversions: 2.2148 MiB (Also required by: active_support/core_ext/time, active_support/core_ext/date_time/conversions)
active_support/values/time_zone: 2.1602 MiB (Also required by: active_support/time_with_zone, active_support/core_ext/date_time/conversions)
tzinfo: 1.8984 MiB
active_support/core_ext/array/conversions: 0.6211 MiB (Also required by: active_support/duration, action_dispatch/http/request, and 2 others)
active_support/xml_mini: 0.5508 MiB (Also required by: active_support/core_ext/hash/conversions)
active_support/core_ext/object/json: 1.0352 MiB (Also required by: active_support/json/encoding)
json: 0.7227 MiB (Also required by: active_support/json/decoding, pg/text_encoder, and 19 others)
json/common: 0.3906 MiB (Also required by: json/ext)
active_support: 0.7109 MiB (Also required by: active_support/time, active_support/railtie, and 5 others)
active_support/dependencies/autoload: 0.332 MiB (Also required by: rails, active_support/rails)
active_support/inflector/methods: 0.332 MiB (Also required by: active_support/core_ext/string/inflections, active_support/core_ext/time/conversions, and 5 others)
active_support/inflections: 0.3164 MiB (Also required by: active_support/inflector)
active_support/logger: 0.3281 MiB
action_dispatch/railtie: 0.668 MiB (Also required by: action_controller/railtie)
action_dispatch: 0.6289 MiB (Also required by: action_controller)
active_support/rails: 0.4414 MiB (Also required by: active_record, active_model, and 4 others)
active_support/deprecation: 0.418 MiB (Also required by: sass/rails/helpers, formtastic/deprecation)
active_record/railtie: 2.8555 MiB
active_record: 1.9766 MiB (Also required by: friendly_id)
active_record/connection_adapters/abstract_adapter: 0.8516 MiB
arel: 0.8086 MiB (Also required by: active_record/base)
arel/visitors: 0.3438 MiB
action_controller/railtie: 0.875 MiB (Also required by: rails/all, sprockets/railtie)
action_controller: 0.6797 MiB (Also required by: heroku_rails_deflate/serve_zipped_assets)
action_controller/metal/live: 0.332 MiB
sprockets/railtie: 1.8516 MiB (Also required by: sass/rails/railtie)
sprockets/rails/helper: 0.918 MiB
action_view/helpers: 0.8945 MiB (Also required by: action_view/base)
action_view/helpers/form_helper: 0.3828 MiB (Also required by: action_view/helpers/form_options_helper)
sprockets: 0.9102 MiB (Also required by: sprockets/rails/helper)
sprockets/directive_processor: 0.5586 MiB
yaml: 0.5078 MiB (Also required by: active_support/ordered_hash, hirb, and 7 others)
psych: 0.5078 MiB
rails3-jquery-autocomplete: 9.9102 MiB
action_controller/base: 8.4922 MiB
action_controller/metal: 1.8281 MiB
abstract_controller/base: 1.0898 MiB
erubis: 0.8477 MiB (Also required by: action_view/template/handlers/erb, heroics)
erubis/engine: 0.4063 MiB (Also required by: erubis/engine/eruby)
action_dispatch/middleware/stack: 0.6367 MiB
active_support/dependencies: 0.5703 MiB (Also required by: abstract_controller/helpers, active_record/base)
abstract_controller/rendering: 1.5625 MiB
action_view/view_paths: 1.4531 MiB (Also required by: action_view/rendering)
action_view/base: 1.3477 MiB
action_view/lookup_context: 0.5156 MiB
action_view/template: 0.3789 MiB (Also required by: action_view/template/resolver)
action_controller/metal/redirecting: 1.0586 MiB
action_controller/metal/rack_delegation: 0.9297 MiB
action_dispatch/http/request: 0.8555 MiB
action_controller/metal/request_forgery_protection: 0.5859 MiB
action_controller/metal/params_wrapper: 0.5 MiB
action_dispatch/http/mime_type: 0.375 MiB (Also required by: abstract_controller/collector, jbuilder/jbuilder_template)
action_controller/metal/url_for: 0.3008 MiB
rails3-jquery-autocomplete/formtastic: 1.2578 MiB
formtastic/inputs/base: 0.8125 MiB
fog: 8.6523 MiB
fog/vcloud_director: 1.0156 MiB
fog/vcloud_director/compute: 0.9961 MiB
fog/vcloud_director/query: 0.4453 MiB
pp: 0.418 MiB (Also required by: scout_apm)
fog/openstack: 0.8398 MiB
fog/cloudstack: 0.5859 MiB
fog/cloudstack/compute: 0.5859 MiB
fog/internet_archive: 0.543 MiB
fog/internet_archive/storage: 0.5352 MiB
fog/internet_archive/core: 0.3125 MiB
fog/rackspace: 0.3867 MiB
fog/cloudsigma: 0.3594 MiB
fog/cloudsigma/compute: 0.3047 MiB
fog/vcloud: 0.3203 MiB
fog/vcloud/compute: 0.3125 MiB
fog/aws: 0.3125 MiB
twitter: 8.3711 MiB
addressable/uri: 4.2422 MiB (Also required by: twitter/base, twitter/rest/request, and 8 others)
addressable/idna: 2.7891 MiB
addressable/idna/pure: 2.7461 MiB
public_suffix: 0.3242 MiB
twitter/streaming/client: 1.7266 MiB
twitter/streaming/response: 1.4453 MiB
http: 1.418 MiB
http/response: 1.2109 MiB
http/cookie_jar: 0.9219 MiB
http/cookie: 0.9102 MiB
domain_name: 0.8203 MiB
domain_name/etld_data: 0.7813 MiB
twitter/configuration: 0.8477 MiB (Also required by: twitter/rest/help)
twitter/base: 0.6016 MiB (Also required by: twitter/entity, twitter/identity, and 12 others)
twitter/null_object: 0.4688 MiB (Also required by: twitter/trend_results)
naught: 0.4063 MiB
twitter/cursor: 0.8164 MiB (Also required by: twitter/rest/utils, twitter/rest/friends_and_followers, and 2 others)
twitter/rest/request: 0.7852 MiB (Also required by: twitter/rest/utils, twitter/rest/friends_and_followers, and 8 others)
faraday: 0.6406 MiB (Also required by: twitter/rest/client, twitter/rest/response/parse_json, and 6 others)
twitter/rest/client: 0.418 MiB
twitter/rest/api: 0.3828 MiB
asset_sync: 4.9609 MiB
asset_sync/multi_mime: 4.5039 MiB
mime/types: 4.4883 MiB (Also required by: mime/types/columnar)
mime/types/registry: 4.1914 MiB
asset_sync/storage: 0.4375 MiB
fog/core: 0.4063 MiB (Also required by: fog, fog/xml, and 48 others)
platform-api: 4.5273 MiB
platform-api/client: 2.8359 MiB
newrelic_rpm: 3.875 MiB
new_relic/control: 3.875 MiB
new_relic/agent: 3.4883 MiB
new_relic/agent/agent: 1.7422 MiB
new_relic/agent/configuration/manager: 0.4961 MiB (Also required by: new_relic/agent/configuration)
new_relic/agent/configuration/default_source: 0.3633 MiB
new_relic/agent/transaction_sampler: 0.3516 MiB
carrierwave: 2.8281 MiB (Also required by: carrierwave-aws)
carrierwave/uploader: 1.7656 MiB
friendly_id: 2.5273 MiB
friendly_id/object_utils: 2.4453 MiB
active_record/base: 2.4258 MiB
active_record/querying: 0.3906 MiB
compass: 2.1328 MiB (Also required by: compass-rails)
compass/sass_extensions: 1.2461 MiB
compass/sass_extensions/sprites: 0.6875 MiB
compass/sass_extensions/sprites/engines: 0.4609 MiB
compass/sass_extensions/sprites/engines/chunky_png_engine: 0.4492 MiB
chunky_png: 0.4492 MiB
compass/sass_extensions/functions: 0.5078 MiB
compass/configuration: 0.6172 MiB
compass/configuration/data: 0.6133 MiB
pg: 2.1055 MiB
pg_ext: 1.9102 MiB
feedjira: 2.0781 MiB
loofah: 0.625 MiB
sax-machine: 0.4414 MiB
sass-rails: 1.8047 MiB
sass/rails: 1.8008 MiB
sass/rails/helpers: 1.7578 MiB
sprockets/sass_functions: 1.7383 MiB
sass: 1.7383 MiB (Also required by: sass/rails/importer, sprockets/sass_importer)
sass/engine: 1.4492 MiB
sass/script: 0.5586 MiB (Also required by: sass/script/css_parser)
sass/scss: 0.3281 MiB
sass/scss/parser: 0.3047 MiB
scout_apm: 1.418 MiB
webrick: 0.5664 MiB
jbuilder: 1.1016 MiB (Also required by: jbuilder/jbuilder_template)
active_support/cache: 0.3906 MiB
rmagick: 1.0234 MiB
rmagick_internal.rb: 1.0195 MiB
RMagick2.so: 0.6875 MiB
lazyload-rails: 0.8555 MiB
nokogiri: 0.8555 MiB (Also required by: TOP, sax-machine/handlers/sax_nokogiri_handler, and 5 others)
sitemap_generator: 0.5195 MiB
dalli: 0.4648 MiB
unicorn: 0.457 MiB
hirb: 0.3984 MiB
I imagine 92 Mb is quite a bit in terms of gems but I do use them all in the app. Carrierwave is the largest, but very important in how I cache images on AWS (as well as with Fog) and so on. This only accounts for about 1/4 of the memory used at boot though.
In my application controller I have a before filter where I load some variables that are used throughout the website (in global headers, sidebars etc) like this:
@search = TextSearch.new
@administrator_logged_in = current_administrator
@nbr_of_stores_total = Store.active.size # Used in header
@nbr_of_blog_posts_total = BlogPost.once_posted.size # Used in header
contests_open = Contest.active.open.size
@active_contests_string = contests_open > 0 ? " (" << contests_open.to_s << ")" : ""
add_breadcrumb "Home", root_path
@open_graph_image = "https://s3.eu-central-1.amazonaws.com/myapp/assets/myapp_logo_large.png"
# Season-related
@is_christmas = false
@gift_word_seasonal = "christmas presents"
I would imagine this would be loaded into boot (and overall) memory, right?
Where else, and how, should I look for startup memory-usage? In short:
Where in my rails app should I trouble boot memory (like middleware, initializers etc)?
What tools (such as the derailed command above) can I use to find/pinpoint memory hungry calls/actions?
I am having trouble finding the direct cause and effect on my trouble shooting actions. Like, finding the answer to "if I remove this code, how much memory do I save?". How should I measure/benchmark any actions I do on my app (rather than upload and check the overall usage in the graph)?
641
Try running GC.stat command This would give info on memory usage and garbage collection in the app based on which you can decide.
Do memory profiling https://github.com/ice799/memprof could also be of help to you
192
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
