Orchestration engines and frameworks?

Question

I'm looking for an orchestration framework/engine/toolkit with which to replace/upgrade an existing software, mainly because of scalability limitations. By orchestration I mean asynchronous and distributed execution of generic tasks and workflows.

More specifically the requirements are pretty much these:

• Wrapping and execution of generic tasks, in Java if language dependent
• API for tasks and workflows on-demand triggering
• Scheduling would be nice as well
• Support for distributed architecture & scalability (mainly for big numbers of small tasks)
• Persistency and resilience
• Advanced workflow configuration capabilities (do this, then these 3 tasks in parallel, then this, having priorities, dependencies...)
• Monitoring and administration UI (or at least API)

The existing system is an old fashion monolithic service (in Java) that has most of that, including the execution logic itself which should remain as untouched as possible.

Does anyone have experience with a similar problem? It seems to me it's supposed to be pretty common, would be strange if I have to implement it myself entirely. I found some questions here (like this and this) discussing the theory of orchestration and choreography systems, but not real examples of tools implementing it. Also I think we're not exactly talking about microservices - the tasks are not prolonged and heavy, they're just many, running in the background executing short jobs of many types. I wouldn't create a service for every job type.

I'm also not looking for cloud and container services at this point - to my understanding the deployment is a different issue.

The closest I got is the Netflix Conductor engine, which answers most of the requirements by running an orchestration server that manages tasks implemented in servlets (or any web services in any language - a plus). However it seems like it's built mainly for arranging heavy tasks in a workflow rather than running a huge number of small tasks, which makes me wonder what would be the overhead of invoking many small tasks in servlets for example.

Does anyone have experience or any input on the Conductor or other tools I could use? Or even my entire approach to the problem?

EDIT: I realize it's kind of a "research advice needed" so let's put it simply in 3 questions:

1. Am I right to look for an orchestration solution for the requirements above?
2. Does anyone have experience with the Netflix Conductor? Any feedback on it?
3. Does it have good competitors?

Answer 1

The main competitor of Netflix Conductor is Temporal Workflow. It scales better and is more developer-friendly by using code instead of JSON DSL to implement the orchestration logic.

It also works OK with the fine-grained tasks by implementing specific optimizations (local activities) that allow batching multiple small tasks into a single database update.

Temporal has been production hardened for over five years at Uber, Coinbase, HashiCorp, Dagadog, Stripe, and hundreds of other companies.