Laravel Nested Jobs

Question

I created a job that has a foreach loop that dispatches another job. Is there a way to fire an even when all the nested jobs are completed?

When triggered here is what happends

Step 1. first I trigger the batch job GenerateBatchReports::dispatch($orderable);

Step 2. We then run a loop and queue other jobs

/**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $dir = storage_path('reports/tmp/'.str_slug($this->event->company) . '-event');

        if(file_exists($dir)) {
            File::deleteDirectory($dir);
        }

        foreach($this->event->participants as $participant) {
            $model = $participant->exercise;

            GenerateSingleReport::dispatch($model);
        }
    }

I just need to know when all the nested jobs are done so I can zip the reports up and email them to a user. When the batch job is done queueing all the nested jobs, it is removed from the list. Is there a way to keep the job around until the nested jobs are done, then fire an event?

Any help would be appreciated.

Answer 1

Update: Laravel 8 (released planned on 8th September 2020) will provide jobs batching. This feature is already documented is probably perfect for nested jobs scenario and looks like this:

$batch = Bus::batch([
    new ProcessPodcast(Podcast::find(1)),
    new ProcessPodcast(Podcast::find(2)),
    new ProcessPodcast(Podcast::find(3)),
    new ProcessPodcast(Podcast::find(4)),
    new ProcessPodcast(Podcast::find(5)),
])->then(function (Batch $batch) {
    // All jobs completed successfully...
})->catch(function (Batch $batch, Throwable $e) {
    // First batch job failure detected...
})->finally(function (Batch $batch) {
    // The batch has finished executing...
})->dispatch();

We will also be able to add additional batched jobs on the fly:

$this->batch()->add(Collection::times(1000, function () {
    return new ImportContacts;
}));

Original answer 👇

I came up with a different solution, because I have a queue using several processes. So, for me:

• No dispatchNow because I want to keep jobs running in parallel.
• Having several processes, I need to make sure the last nested job will not run after the final one. So a simple chaining doesn’t guarantee that.

So my not elegant solution filling the requirements is to dispatch all the nested jobs and, in the last one, dispatching the final job with a couple of seconds of delay, to make sure all other nested jobs that may still be running in parallel will be terminated.

/**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $last_participant_id = $this->event->participants->last()->id;

        foreach($this->event->participants as $participant) {
            $is_last = $participant->id === $last_participant_id;

            GenerateSingleReport::dispatch($model, $is_last);
        }
    }

and in GenerateSingleReport.php

class GenerateSingleReport implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $model;
    protected $runFinalJob;

    public function __construct($model, $run_final_job = false)
    {
        $this->model = $model;
        $this->runFinalJob = $run_final_job;
    }

    public function handle()
    {
        // job normal stuff…

        if ($this->runFinalJob) {
            FinalJob::dispatch()->delay(30);
        }
    }
}

Alternatively

I’m throwing another idea, so the code is not flawless. Maybe a wrapper Job could be created and dedicated to running the last nested job chained with the final job.

/**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $last_participant_id = $this->event->participants->last()->id;

        foreach($this->event->participants as $participant) {
            $is_last = $participant->id === $last_participant_id;

            if ($is_last) {
                ChainWithDelay::dispatch(
                    new GenerateSingleReport($model), // last nested job
                    new FinalJob(), // final job
                    30 // delay
                );
            } else {
                GenerateSingleReport::dispatch($model, $is_last);
            }
        }
    }

And in ChainWithDelay.php

class ChainWithDelay implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $job;
    protected $finalJob;
    protected $delay;

    public function __construct($job, $final_job, $delay = 0)
    {
        $this->job = $job;
        $this->finalJob = $final_job;
        $this->delay = $delay;
    }

    public function handle()
    {
        $this->job
            ->withChain($this->finalJob->delay($this->delay))
            ->dispatchNow();
    }
}

Answer 2

For laravel >= 5.7

You can use the dispatchNow method. That will keep the parent job alive while the child jobs are processing:

https://laravel.com/docs/5.8/queues#synchronous-dispatching

Parent job:

public function handle()
{
    // ...

    foreach($this->event->participants as $participant) {
        $model = $participant->exercise;

        GenerateSingleReport::dispatchNow($model);
    }

    // then do something else...
}

For laravel 5.2 - 5.6

You could use the sync connection:

https://laravel.com/docs/5.5/queues#customizing-the-queue-and-connection

Make sure the connection is defined in your config/queue.php:

https://github.com/laravel/laravel/blob/5.5/config/queue.php#L31

Parent job (NOTE: This syntax is for 5.5. The docs are a little different for 5.2):

public function handle()
{
    // ...

    foreach($this->event->participants as $participant) {
        $model = $participant->exercise;

        GenerateSingleReport::dispatch($model)->onConnection('sync');
    }

    // then do something else...
}

Answer 3

You could use Laravel's job chaining. It allows you to run a bunch of jobs in sequence and if one fails, the rest in the chain will not be run.

The basic syntax looks like this:

FirstJob::withChain([
    new SecondJob($param),
    new ThirdJob($param)
])->dispatch($param_for_first_job);

In your case your could add all of your GenerateSingleReport jobs to an array except the first one and add then add the final job that you want to run to the end of the array. Then you can pass that array to the withChain method on the first job.

$jobs = [];
$first_job = null;
$first_parameter = null;

foreach($this->event->participants as $participant) {
    $model = $participant->exercise;

    if (empty($first_job)) {
        $first_job = GenerateSingleReport;
        $first_parameter = $model;
    } else {
        $jobs[] = new GenerateSingleReport($model);
    }            
}

$jobs[] = new FinalJob();

$first_job->withChain($jobs)->dispatch($first_parameter);