Is there Jenkins or other open-source CI for a distributed system?

Question

I have a set of stateful services that compose a distributed system. Each one has a Docker file that can build it and run its unit tests. I need to check how they interact with each other in a distributed manner. To run tests, I need to:

• Be able to run multiple Docker files at the same time on different nodes that would be considered a single test run.
• Be able to set barriers in commands execution order of each Docker file. In other words, assuming all Docker files run simultaneously I need to be sure that all files have finished stage X before starting stage Y.
• Be able to stop on failure of any node.

Is there a Jenkins plugin/extension for such operations, or is there any other system or approach that could solve such problem?

Answer 1

Use gitlab and gitlab-ci with autoscale,

this is supported out of the box, see autoscale runners Gitlab-CI which uses Docker Machine with drivers to automatic deploy resources as needed, even on digital ocean.

And you describe your build and tests in the .gitlab-ci.yml:

• defining the different stages, (build, test1 .... )
• where all builds/tests per stage have to finish before moving to the next stage
• and per stage all run parallel, depending on availleble resources of course.

see my simple docker pipeline here.

REMARK 1: When confidentiality is an issue, you can

• a. create a private repository on gitlab
• b. host gitlab on your own server*

REMARK 2: When code is hosted somewhere else (e.g. github), you can push it to gitlab on every commit to initiate the build pipeline.

If you would need a professional to setup such a project, you can contact me, but that would be outside the scope of this question/answer