Agenda jobs, is it possible to schedule a job to be triggered after the previous one finished?

Question

I'm using agenda jobs in nodejs. I was wondering if it's possible to define a job with a dependency. In my case I have 3 jobs, each one of the jobs execute a part of a whole logic, which is quite big, that's why they are separated.

1. In the Job 1, I collect info from the database, transform them and perform a few tasks.
2. In the Job 2, I get all the data that had been previously processed by the Job 1 and perform new transformations..
3. In the Job 3, I get again all the data that had been processed in the Job 2 and send a few reports.

I have those tasks schedule to be executed every 5 minutes. But the thing is that the 3 of them are schedule at the same time because, they are schedule dynamically.

job1.schedule("now");
job1.repeatEvery('5 minutes');

job2.schedule("now");
job2.repeatEvery('5 minutes');

job3.schedule("now");
job3.repeatEvery('5 minutes');

As it's configured currently, to process one instance that needs to go through the 3 jobs, worst case scenario the user will need to wait 15min, which is not ideal.

I was wondering if there is an option to define that the tasks should be executed as soon as the previous task is finished. I'm aware that a possible workaround will be to schedule the jobs with a few minutes of difference, but given that depending on the number of instances the job can require more or less time this doesn't work for me.

Answer 1

The way I am doing a pretty similar thing is, I trigger the next job when the main function of the preceding job is done, this way I can feed data from one job to another and run them in sequence.

So I trigger job3 in the definition of job2 and trigger job2 in the definition of job1:

// DEFINE JOBS
agenda.define('job1', (job) => {
    return myMainJob1Function(job.attrs.data)
      .then((data) => {
        for (const element of data) {
          agenda.now('job2', { // runs many instances of 'job2' job with distinct data
            // use output data from the "parent" job
            arg1: element.arg1,
            arg2: element.arg2,
            arg3: element.arg3,
            arg4: job.attrs.data.someArg, // this will propagate an argument we set for the "parent" job
          });
        }
      });
  });

agenda.define('job2', (job) => {
    return myMainJob2Function(job.attrs.data) // data supplied from 'job1'
      .then((data) => {
        for (const element of data) {
          agenda.now('job3', { // runs many instances of 'job3' job with distinct data
            // use output data from the "parent" job
            arg1: element.arg1,
            arg2: element.arg2,
            arg3: element.arg3,
            arg4: job.attrs.data.someArg, // this will propagate an argument we set for the "parent" job
          });
        }
      });
  });

agenda.define('job3', (job) => {
    return myMainJob3Function(job.attrs.data) // data supplied from 'job2'
      .then((data) => {
        // save data to database or do something else
      });
  });

// TRIGGER 'job1', WHICH TRIGGERS CHILD JOBS
agenda.now('job1', { someArg: 5 }); // we're providing some initial argument, that gets passed onto child jobs
const schedule = '0 0/5 0 ? * * *' // cron expression for every 5 minutes
agenda.every(schedule, 'job1', { someArg: 5 });

One other thing that comes to mind is using triggers for complete or success of job1 but when you run many of the same jobs with different input data you can't (I haven't found a way) listen to those events with regard to the specific job instance that you want.

That being said you can do:

agenda.on('success:job1', (job) => {
  agenda.now('job2', { someArg: job.attrs.data.someArg });
});