How to manage/balance semi persistent jobs over service instances

Question

I see a common pattern for services that we try to develop and I wonder if there are tools / libraries out there that would help here. While the default jobs as discussed in microservice literature is from the REQUEST -> RESPONSE nature, our jobs are more or less assignments of semi permanent tasks.

Examples of such tasks

• Listen on the message queue for data from source X and Y, correlate the data that comes in and store it in Z.
• Keep an in-memory buffer that calculates a running average of the past 15 mins of data everytime a new data entry comes in.

Currently our services are written in PHP. Due to the perceived overhead of PHP processes and connections to the message queue we'd like a single service process to handle multiple of those jobs simultanously.

A chart that hopefully illustrated the setup that we have in our head:

• Service Workers are currently deamonized PHP scripts
• For the Service Registry we are looking at Zookeeper

While Zookeeper (and Curator) do loadbalancing, I did not find anything around distributing permanent jobs (that are updatable, removable, and must be reassigned when a worker dies)

Proposed responsibilities of a Job Manager

• Knows about jobs
• Knows about services that can do these jobs
• Can assign jobs to services
• Can send job updates to services
• Can reassign jobs if a worker dies

Are there any libraries / tools that can tackle such problems, and can thus function as the Job Manager? Or is this all one big anti pattern and should we do it some other way?

Answer 1

You should have a look at Gearman.

It composes of a client which assigns the jobs, one or more workers which will pick up and execute the jobs and a server which will maintain the list of functions (services) and jobs pending. It will re-assign the jobs if a worker dies.