How to catch Memory Quota exceptions in a heroku worker

Question

I am using delayed_job to handle my background jobs with heroku. Occasionally I will overstep my memory allocation and I get stuff like:

2011-11-16T02:41:25+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded) 2011-11-16T02:41:45+00:00 heroku[worker.1]: Process running mem=542M(106.0%)

I would like to elegantly handle this. Is there a way to find out when I am about to cross my memory limit?

Something like rack-timeout would be awesome

Thanks!

Answer 1

I think I found a good solution to this, by stealing some code from the Oink gem. In particular this file: memory_snapshot.rb, Which you should read. It outlines 4 different ways to messure memory usage

So there is no way to have this on the Rack level, you need to add a memory check during a process that is causing the memory issue (in my case it was building a csv file).

So in that loop it looked something like:

  def build_string_io(collection)
    csv_io = StringIO.new
    csv_io << collection.first.to_comma_headers.join(',') + "\n"
    collection.each do |imp|
      csv_io << imp.to_comma.join(',') + "\n"
      check_memory!
    end
    csv_io.rewind
    csv_io
  end

  def check_memory!
    raise 'AboutToRunOutOfMemory' if memory > 400.megabytes #Or whatever size your worried about
  end


  # Taken from Oink
  def memory
     pages = File.read("/proc/self/statm")
     pages.to_i * self.class.statm_page_size
  end

  def self.statm_page_size
      @statm_page_size ||= begin
      sys_call = SystemCall.execute("getconf PAGESIZE")
      if sys_call.success?
        sys_call.stdout.strip.to_i / 1024
      else
        4
      end
    end
  end

Answer 2

The problem you have is that you need the data which is only available from the logs.

The best approach here will be to use a syslog drain to send your logs to a service like Papertrailapp.com or Loggly - with these services you can set up searches for your R14 error but then receive notifications - Papertrail supports campfire, posts, emails etc where you can handle the error.

We do this exact process posting into a sinatra app also hosted on Heroku where we're looking at the heroku router log entries and the queue= sizes or for backlog too deep errors and then automatically scaling our applications when demand requires - because the syslog is almost realtime our apps are essentially self aware.