Detect which Sidekiq jobs are responsible for high memory usage

Question

I run an app on Heroku and use Sidekiq as the job queue system. However, recently, memory usage is always around 90%-110%.

I already tried reducing the concurrency a little and scaling the number of workers, but with no big success. Is there a way to detect which Sidekiq jobs are consuming so much memory?

We use New Relic to track our transactions but I couldn't find this information on the platform.

Answer 1

I faced the similar situation where I need to track memory used by workers and I have come up with the following solution, Not sure if it can help you, But I hope it'll set the right direction to find the solution

I have written a cron job which collects currently running workers(not realtime) and memory usage and stores it into a csv file. Below is the code.

class DataWorker
  include Sidekiq::Worker

  def perform
    file = File.new("sidekiq_profiling.csv", "a")
    memory_usage = gc_start
    workers = Sidekiq::Workers.new
    worker_running_counts = workers.map {|pid, thrd, wrkr| wrkr["payload"]["class"]}.group_by {|cls| cls}.map {|k, v| {k => v.count}}
    datetime = DateTime.now
    worker_running_counts.each do |wc|
      file << "#{datetime},#{wc.keys[0]},#{wc.values[0]},#{memory_usage}\n"
    end
    file.close
  end


  def rss_usage
    `ps -o rss= -p #{Process.pid}`.chomp.to_i * 1024
  end

  # def gc_stats
  #   GC.stat.slice(:heap_available_slots, :heap_live_slots, :heap_free_slots)
  # end

  def gc_start
    GC.start
    # gc_stats.each do |key, value|
    #   puts "GC.#{key}: #{value.to_s(:delimited)}"
    # end
    "#{rss_usage.to_s(:human_size, precision: 3)}"
  end

  Sidekiq::Cron::Job.create(name: 'DataWorker', cron: '* * * * *', class: 'DataWorker')
end