How long should a Sidekiq job last?

Question

In the Sidekiq wiki it is stated:

1. Make your jobs small and simple

I get simple, I get idempotent and transactional, but what is small? Maybe required Memory and Computing Time is a good measure? My Sidekiq jobs take between 10sec and 30min.

I think that 10sec is okay, but what about the long-running task of 30min? I am loading all the data of a certain type from the database into memory, run lengthy computations on them and then write back the results. All three things in one worker job.

Is that fine? Or should I instead invoke from a worker job, multiple worker jobs that run the small computations? The problem is, these small computations may need some complex hash tables to do the computations and it was suggested not to persist this in Redis, only small simple values.

Answer 1

That depends on how often you want/have to invoke the job and whether it is acceptable to you that it takes so long.

If you run the job in shorter intervals than it takes to finish it certainly is too long.

Splitting it up into multiple workers would only help here if you could improve the total run time (e.g. if some of it can be run at the same time)

So rule of thumb as always: as long as it suits your needs it is ok.

However:

• on long jobs you should consider that the job might fail mid-execution for whatever reason (server crashes etc).
• Can you still continue this lengthy job there or will it be rolled back properly?
• Also: What happens if data is changed while you are executing the job?