Laravel run multiple scheduled tasks

Question

I currently have a scheduled console command that runs every 5 minutes without overlap like this:

 $schedule->command('crawler')
             ->everyFiveMinutes()
             ->withoutOverlapping()
             ->sendOutputTo('../_laravel/storage/logs/scheduler-log.txt');

So it works great, but I currently have about 220 pages that takes about 3 hours to finish in increments of 5 minutes because I just force it to crawl 10 pages at each interval since each page takes like 20-30 seconds to crawl due to various factors. Each page is a record in the database. If I end up having 10,000 pages to crawl, this method would not work because it would take more than 24 hours and each page is supposed to be re-crawled once a day.

So my vendor allows up to 10 concurrent requests (or more with higher plans), so what's the best way to run it concurrently? If I just duplicate the scheduler code, does it run the same command twice or like 10 times if I duplicated it 10 times? Any issues that would cause?

And then I need to pass on parameters to the console such as 1, 2, 3, etc... in which I could use to determine which pages to crawl? i.e. 1 would be 1-10 records, 2 would be next 11-20 records, and so on.

Using this StackOverfow answer, I think I know how to pass it along, like this:

 $schedule->command('crawler --sequence=1')

But how do I read that parameter within my Command class? Does it just become a regular PHP variable, i.e. $sequence?

Answer 1

1. Better to use queue for job processing
2. on cron, add all jobs to queue
3. Run multiple queue workers, which will process jobs in parallel

Tip: It happened with us. It might happen that job added previously is not complete, yet cron adds the same task in queue again. As queues works sequentially. To save yourself from the situation, you should in database mark when a task is completed last time, so you know when to execute the job (if it was seriously delayed)

Answer 2

I found this on the documentation, I hope this is what you're looking for:

• Retrieving Input

While your command is executing, you will obviously need to access the values for the arguments and options accepted by your application. To do so, you may use the argument and option methods:

• Retrieving The Value Of A Command Argument

$value = $this->argument('name');

• Retrieving All Arguments

$arguments = $this->argument();

• Retrieving The Value Of A Command Option

$value = $this->option('name');

• Retrieving All Options

$options = $this->option();

source