Rails - How do I prevent Sidekiq from slowing down the server?

Question

I have a newsletter that I send out to my customers (~10k emails) every morning and sometimes happens that this Sidekiq job is taking some much CPU/memory performance that the website (Rails app) is not running and facing blackouts.

When I look at the Sidekiq dashboard, I see there is some problem (probably invalid email address and Sidekiq repeatedly trying to send it again?) with the newsletter and it's stuck.

How do I prevent this behavior and preclude repeating the Sidekiq task (which I believe that's the problem of the breakout)?

Here's my code:

rake task:

namespace :mailer do  desc "Carrier blast - morning"
  task :newsletter_morning => [:environment] do                                            
    NewslettertJob.perform_later
  end
end

job definition:

class NewslettertJob < ApplicationJob
  def perform
    ...
    NewsletterMailer.morning_blast(data).deliver_now
  end
end

and NewsletterMailer:

class NewsletterMailer < ApplicationMailer
  def morning_blast(data)
    ...
    customers.each do |customer|
      yield customer, nil; next if customer.email.blank?

      begin
        Retryable.retryable( tries: 1, sleep: 30, on: [Net::OpenTimeout, Net::SMTPAuthenticationError, Net::SMTPServerBusy]) do
          send_email(customer.email).deliver
        end
        send_email(customer.email).deliver
      rescue Net::SMTPSyntaxError => e
        error_msg = "Newsletter sending failed on #{Time.now} with: #{e.message}. e.inspect: #{e.inspect}"
        logger.warn error_msg
        yield customer, nil
        next
      end
    end
  end
end

What I want to achieve is that the newsletter will be sent out every morning and if Rails/Sidekiq faces a problem, it will simply shut itself down, so the newsletter will not affect the "life" on the main website (its server).

Thank you in advance for every advice. I am being stuck on this issue for a while now.

Answer 1

1. If your machine only has one core, Sidekiq and puma will fight for CPU. Lower Sidekiq's concurrency so it uses less CPU, or get a machine with multiple cores, or move Sidekiq to a different machine.

2. If a Sidekiq process is using 100% of a core, lower the concurrency setting. The default in Sidekiq 6.0 is 10, which is a good default but if you are just delivering emails you could probably bump that to 20. You can run multiple Sidekiq processes if you wish to utilize multiple cores to process jobs faster.

Answer 2

I think ideally, you should separate your background task servers from your web servers, that way background process won't impact on the performance of the web server. I work for a very high traffic/ high-load company, and we have an architecture of sorts in here.

There are explanations on how to stop retries in this answer: Disable automatic retry with ActiveJob, used with Sidekiq

Another thing, your e-mail sending is done synchronously (.deliver). This implicates on your task being a huge monolitical process with many customers, with huge impact on memory. Instead, you could use a deliver_later, so each customer get's it's own little worker. This will also help aliviate CPU and Memory usage. You could even create a worker for sending e-mails per customer, and use your monolitical Job to merely dispatch those.

class NewslettertJob < ApplicationJob
  def perform
    ...
    customers.each |customer| do
      NewsletterMailer.morning_blast(customer, data).deliver_later if customer.email.present?
    end 
  end
end

However, I think the silver bullet is separating your sidekiq server from your web server - having one server dedicated to background tasks. On your web server, you don't even start the sidekiq instances.