Sidekiq - Only handle error after x retries?

Question

I'm using sidekiq to process thousands of jobs per hour - all of which ping an external API (Google). One out of X thousand requests will return an unexpected (or empty) result. As far as I can tell, this is unavoidable when dealing with an external API.

Currently, when I encounter such response, I raise an Exception so that the retry logic will automatically take care of it on the next try. Something is only really wrong with the same job fails over and over many times. Exceptions are handled by Airbrake.

However my airbrake gets clogged up with these mini-outages that aren't really 'issues'. I'd like Airbrake to only be notified of these issues if the same job has failed X times already.

Is it possible to either

• disable the automated airbrake integration so that I can use the sidekiq_retries_exhausted to report the error manually via Airbrake.notify
• Rescue the error somehow so it doesn't notify Airbrake but keep retrying it?
• Do this in a different way that I'm not thinking of?

Here's my code outline

class GoogleApiWorker
  include Sidekiq::Worker
   sidekiq_options queue: :critical, backtrace: 5

  def perform
    # Do stuff interacting with the google API
  rescue Exception => e
    if is_a_mini_google_outage? e
      # How do i make it so this harmless error DOES NOT get reported to Airbrake but still gets retried?
      raise e
    end
  end

  def is_a_mini_google_outage? e
    # check to see if this is a harmless outage
  end    
end

Answer 1

As far as I know Sidekiq has a class for retries and jobs, you can get your current job through arguments (comparing - cannot he effective) or jid (in this case you'd need to record the jid somewhere), check the number of retries and then notify or not Airbrake.

https://github.com/mperham/sidekiq/wiki/API https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/api.rb

(I just don't give more info because I'm not able to)

Answer 2

Sidekiq supports a sidekiq_retries_exhausted hook that you can setup for your job if you want to, for example, log an entry or update a status on a model if all the retries failed:

class FailingJob
  include Sidekiq::Job
  sidekiq_options retry: 3

  sidekiq_retries_exhausted do |job, ex|
    Sidekiq.logger.warn "Failed #{job['class']} with #{job['args']}: #{job['error_message']}"
  end
  
  def perform(*args)
    raise "or I don't work"
  end
end