Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Running stages in parallel with Jenkins workflow / pipeline

Please note: the question is based on the old, now called "scripted" pipeline format. When using "declarative pipelines", parallel blocks can be nested inside of stage blocks (see Parallel stages with Declarative Pipeline 1.2).

I'm wondering how parallel steps are supposed to work with Jenkins workflow/pipeline plugin, esp. how to mix them with build stages. I know about the general pattern:

parallel(firstTask: {   // Do some stuff }, secondTask: {   // Do some other stuff in parallel }) 

However, I'd like to run couple of stages in parallel (on the same node, which has multiple executors), so I tried to add stages like this:

stage 'A' // Do some preparation stuff  parallel(firstTask: {   stage 'B1'   // Do some stuff }, secondTask: {   stage 'B2'   // Do some other stuff in parallel })  stage 'C' // Finalizing stuff 

This does not work as expected. The "do stuff" tasks are executed in parallel, but the parallel stages end immediately and do not incorporate the stuff they should contain. As a consequence, the Stage View does not show the correct result and also does not link the logs.

Can I build different stages in parallel, or is the "parallel" step only meant to be used within a single stage?

like image 344
ami Avatar asked Apr 26 '16 18:04

ami


People also ask

Can Jenkins run parallel builds?

Jenkins is a great CI tool when it comes to flexibility. Every simple thing can be done in ten different ways, including parallelizing a build.

How does parallel execution work in Jenkins?

This plugin adds a tool that lets you easily execute tests in parallel. This is achieved by having Jenkins look at the test execution time of the last run, split tests into multiple units of roughly equal size, then execute them in parallel.

How do you run the same job multiple times in parallel with Jenkins?

You will need to configure your "child" job so that it can run in parallel by checking the "Execute concurrent builds if necessary" in the job configuration. Whatever set of slaves provide the connection to the embedded devices will need enough executors to run your jobs in parallel.


1 Answers

You may not place the deprecated non-block-scoped stage (as in the original question) inside parallel.

As of JENKINS-26107, stage takes a block argument. You may put parallel inside stage or stage inside parallel or stage inside stage etc. However visualizations of the build are not guaranteed to support all nestings; in particular

  • The built-in Pipeline Steps (a “tree table” listing every step run by the build) shows arbitrary stage nesting.
  • The Pipeline Stage View plugin will currently only display a linear list of stages, in the order they started, regardless of nesting structure.
  • Blue Ocean will display top-level stages, plus parallel branches inside a top-level stage, but currently no more.

JENKINS-27394, if implemented, would display arbitrarily nested stages.

like image 77
Jesse Glick Avatar answered Oct 07 '22 17:10

Jesse Glick