How can I tell Sentry not to alert certain exceptions?

Question

I have a Rails 5 application using raven-ruby to send exceptions to Sentry which then sends alerts to our Slack.

Raven.configure do |config|
  config.dsn = ENV['SENTRY_DSN']
  config.environments = %w[ production development ]
  config.excluded_exceptions += []
  config.async = lambda { |event|
    SentryWorker.perform_async(event.to_hash)
  }
end

class SentryWorker < ApplicationWorker
  sidekiq_options queue: :default

  def perform(event)
    Raven.send_event(event)
  end
end

It's normal for our Sidekiq jobs to throw exceptions and be retried. These are mostly intermittent API errors and timeouts which clear up on their own in a few minutes. Sentry is dutifully sending these false alarms to our Slack.

I've already added the retry_count to the jobs. How can I prevent Sentry from sending exceptions with a retry_count < N to Slack while still alerting for other exceptions? An example that should not be alerted will have extra context like this:

sidekiq: {
  context: Job raised exception,
  job: {
    args: [{...}],
    class: SomeWorker,
    created_at: 1540590745.3296254,
    enqueued_at: 1540607026.4979043,
    error_class: HTTP::TimeoutError,
    error_message: Timed out after using the allocated 13 seconds,
    failed_at: 1540590758.4266324,
    jid: b4c7a68c45b7aebcf7c2f577,
    queue: default,
    retried_at: 1540600397.5804272,
    retry: True,
    retry_count: 2
  },
}

What are the pros and cons of not sending them to Sentry at all vs sending them to Sentry but not being alerted?

Answer 1

Summary

An option that has worked well for me is by configuring Sentry's should_capture alongside Sidekiq's sidekiq_retries_exhausted with a custom attribute on the exception.

Details

1a. Add the custom attribute

You can add a custom attribute to an exception. You can define this on any error class with attr_accessor:

class SomeError
  attr_accessor :ignore

  alias ignore? ignore
end

1b. Rescue the error, set the custom attribute, & re-raise

def perform
  # do something
rescue SomeError => e
  e.ignore = true
  raise e
end

2. Configure should_capture

should_capture allows you to capture exceptions when they meet a defined criteria. The exception is passed to it, on which you can access the custom attribute.

config.should_capture { |e| !e.ignore? }

3. Flip the custom attribute when retries are exhausted

There are 2 ways to define the behaviour you want to happen when a job dies, depending on the version of Sidekiq being used. If you want to apply globally & have sidekiq v5.1+, you can use a death handler. If you want to apply to a particular worker or have less than v5.1, you can use sidekiq_retries_exhausted.

sidekiq_retries_exhausted { |_job, ex| ex.ignore = false }