Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section

Question

Jenkins 2.89.4 rolling

I saw almost all stackoverflow posts which show how we can successfully run parallel steps/stages (using list/maps etc) --OR hardcoding them directly --OR even create dynamic stages for Jenkinsfile (as seen in this post: Scripted jenkinsfile parallel stage)

My requirements are:

1. A pipeline which builds N. no of projects under "BUILD" steps i.e. parallel builds on each of those projects. i.e. it runs Gradle on all N projects. Here I have a Jenkinsfile which was created by a declarative JOB DSL Groovy. Here my Gradle projects are not set as multi-projects so I can't call the top level gradle and say, Gradle please do your parallel magic (within Gradle).

2. I want to run build of these N projects in their own separate parallel dynamically created stages (GUI Columns) as seen in jenkins job's dashboard.

3. I want to see the output of (Gradle build/console) each project's build separately i.e. I don't want to mix the console output of each projects build which are running in parallel in just ONE COLUMN (i.e. column named BUILD).

4. In this URL https://jenkins.io/blog/2017/09/25/declarative-1/ I see, how you can run parallel stages/steps but in doing so, either it's mixing those parallel step's output in just one column (I mean under BUILD column) --OR if you want it under separate stages/columns (i.e. the post says Test on Linux or Windows separately then you are still hard-coding all the stages/steps in Jenkinsfile early on (rather than using just a list or array hash which I'd prefer to update for adding more or less stages / parallel steps as in my case, they all follow the same standard). What I want is to just update in one place How many steps and what are all of the stages in just one place (list/array).

5. I'm not using Jenkins Blue Ocean for now.

Usually if you have parallel steps within a stage, their console std output for all steps gets mixed into one console output / stage/column when you click to see the console output for that given parallel step/stage; When you hover over the BUILD column (assuming there were parallel steps in BUILD stage) in job's dashboard (std output for all those steps is mixed up and very hard to see individual project step's console output just for a given step/stage).

If we want to create separate stages (dynamically) then Jenkins should be able to show console output of a given step/dynamic stage within the parallel section (i.e. each column should show their own project's build console output).

Using the above URL, I'm able to do the following after trying this script:

// main script block
// could use eg. params.parallel build parameter to choose parallel/serial 
def runParallel = true
def buildStages

node('master') {
  stage('Initializing Parallel Dynamic Stages') {
    // Set up List<Map<String,Closure>> describing the builds
    buildStages = prepareBuildStages()
    println("Initialised pipeline.")
  }

  for (builds in buildStages) {
    if (runParallel) {
      parallel(builds)
    } else {
      // run serially (nb. Map is unordered! )
      for (build in builds.values()) {
        build.call()
      }
    }
  }

  stage('Done') {
      println('The whole SHENZI is complete.')
  }
}

// Create List of build stages to suit
def prepareBuildStages() {
  def buildList = []

  for (i=1; i<4; i++) {
    def buildStages = [:]
    for (name in [ 'Alpha', 'Tango', 'Chyarli' ] ) {
      def n = "${name} ${i}"
      buildStages.put(n, prepareOneBuildStage(n))
    }
    buildList.add(buildStages)
  }
  return buildList
}

def prepareOneBuildStage(String name) {
    def proj_name = name.split(' ')[0]    
    def proj_parallel_sub_step = name.split(' ')[1]

    //Return the whole chunkoni
    return {
            stage("Build\nProject-${proj_name}\nStep ${proj_parallel_sub_step}") {
                println("Building ${proj_name} - ${proj_parallel_sub_step}")
                sh(script:'sleep 15', returnStatus:true)
            }
    }
}

When I'm putting the above Groovy Script (which is creating DYNAMIC Stages) inside Pipeline Script or Pipeline Script from SCM (i.e. the same code available in a .groovy file) -- it runs successfully and creates dynamic stages under BUILD step for each of the 3 projects and runs 3 steps (Nth) for all 3 projects in parallel and then starts the next Nth step for all 3 projects and so on.

If you see below, we also got individual columns in Jenkins job dashboard for them.

Now, When I put the above script in Jenkinsfile (Pipeline DSL) where I have pipeline { .... } section, it's not working and giving me the following error.

Using my JOB DSL, I created a new Jenkins Pipeline job where Pipeline Script from SCM calls a groovy file (which now contains):

//----------------------------------------------------
// Both - Parallel Run and GUI View in JF Jenkins job.
//----------------------------------------------------
def runParallel = true
def buildStages
def wkspace = /var/lib/jenkins/workspaces/ignore_this_variale_or_its_value_for_now

// Create List of build stages to suit
def prepareBuildStages() {
  def buildList = []

  for (i=1; i<3; i++) {
    def buildStages = [:]
    for (name in [ 'Alpha', 'Tango', 'Chyarli' ] ) {
      def n = "${name} ${i}"
      buildStages.put(n, prepareOneBuildStage(n))
    }
    buildList.add(buildStages)
  }
  return buildList
}
//---
def prepareOneBuildStage(String name) {
  def proj_name = name.split(' ')[0]
  def proj_parallel_sub_step = name.split(' ')[1]
  // return the whole chunkoni (i.e. for a given stage) - will be named dynamically.
  return {
    stage("Build\nProject-${proj_name}\nStep ${proj_parallel_sub_step}") {
      println("Building ${proj_name} - ${proj_parallel_sub_step}")
      sh(script:'sleep 15', returnStatus:true)
    }
  }
}
// Set up List<Map<String,Closure>> describing the builds
buildStages = prepareBuildStages()
//---------------------




String jenkinsBaselines

// SEE NOW --- we have this section called 'pipeline'
pipeline {
    agent {
        node {
            label 'rhat6'
            customWorkspace wkspace
        }
    }

    options {
        ansiColor('xterm')
        timeout(time: 8, unit: 'HOURS')
        skipDefaultCheckout()
        timestamps()
    }

    environment {
        someEnvVar = 'aValue'

    }

    //------------- Stages
    stages {
        stage('Initializing Parallel Dynamic Stages') {
            // Set up List<Map<String,Closure>> describing the builds
            println("Initialised pipeline.")
        }

        for (builds in buildStages) {
            if (runParallel) {
                parallel(builds)
            } else {
                // run serially (nb. Map is unordered! )
                for (build in builds.values()) {
                    build.call()
                }
            }
        }

        stage('Done') {
            println('The whole SHENZI is complete.')
        }      
    }
    //---------------------
}

Running the Jenkinsfile Jenkins job now gives me this error:

[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 1s
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 69: Not a valid stage section definition: "buildStages = prepareBuildStages()". Some extra configuration is required. @ line 69, column 5.
            stage('Initializing Parallel Dynamic Stages') {
       ^

WorkflowScript: 69: Unknown stage section "println". Starting with version 0.5, steps in a stage must be in a steps block. @ line 69, column 5.
            stage('Initializing Parallel Dynamic Stages') {
       ^

WorkflowScript: 75: Expected a stage @ line 75, column 5.
            for (builds in buildStages) {
       ^

WorkflowScript: 86: Unknown stage section "println". Starting with version 0.5, steps in a stage must be in a steps block. @ line 86, column 5.
            stage('Done') {
       ^

WorkflowScript: 69: No "steps" or "parallel" to execute within stage "Initializing Parallel Dynamic Stages" @ line 69, column 5.
            stage('Initializing Parallel Dynamic Stages') {
       ^

WorkflowScript: 86: No "steps" or "parallel" to execute within stage "Done" @ line 86, column 5.
            stage('Done') {
       ^

6 errors

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:133)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:557)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:518)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:290)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

How can I get this working in Jenkinsfile pipeline section and still able to get individual columns per dynamically created stage for a given project N and Step M?

Tried the following way, still errors the say way.

//------------- Stages
stages {
    stage('Initializing Parallel Dynamic Stages') {
        // Set up List<Map<String,Closure>> describing the builds
        buildStages = prepareBuildStages()
        println("Initialised pipeline.")

        // tried this way too. within a stage
        buildStages.each { bld -->
            parallel(bld)
        }  
    }

    stage('Done') {
        println('The whole SHENZI is complete.')
    }      
}
//---------------------

Answer 1

So I did some poking into this and got it working.

The new code within Jenkinsfile (for STAGES) is now:

//------------- Stages
stages {
     stage('Start Pipeline') {
        steps {
            script {
                sh "echo HELLO moto razr!"
            }
        }  
     }

    stage('Initializing Parallel Dynamic Stages'){
        steps {
            script {
                // Run all Nth step for all Projects in Parallel. 
                buildStages.each { bs -> parallel(bs) }


                // OR uncomment the following code (if conditional on boolean variable).
                /*
                for (builds in buildStages) {
                    if (runParallel) {
                        parallel(builds)
                    } else {
                        // run serially (nb. Map is unordered! )
                        for (build in builds.values()) {
                            build.call()
                        }
                    }
                }
                */        

            }   
        }
     } 

    stage('Done') {
        println('The whole SHENZI is complete.')
    }      
}
//---------------------

That's all it took to work.

For clear messages / stage names, I tweaked the function as well and We won't set this variable buildStages within pipeline { ... }

//---
def prepareOneBuildStage(String name) {
  def proj_name = name.split(' ')[0]
  def proj_parallel_sub_step = name.split(' ')[1]
  // return the whole chunkoni (i.e. for a given stage) - will be named dynamically.
  return {
    stage("BUILD Project-${proj_name} Parallel_Step_${proj_parallel_sub_step}") {
      println("Parallel_Step # ${proj_parallel_sub_step} of Project => ${proj_name}")
      sh(script:"echo \"Parallel_Step # ${proj_parallel_sub_step} of Project => ${proj_name}\" && sleep 20", returnStatus:true)
      // -- OR -- you can call Gradle task i.e. rpm / any other / any other command here. 
    }
  }
}

// Set up List<Map<String,Closure>> describing the builds. section now.
buildStages = prepareBuildStages()
//---------------------

This implementation is now creating N no. of parallel stages i.e. a separate column per project for a given Nth step (when looking at Jenkinsfile job's dashboard) for P no. of projects.

1. It will run All P projects for a given Nth step in parallel.

2. It will wait for Nth step for all Projects to complete first and then jump to the next Nth step.

   What this means is, if Project ALPHA Step #1 is complete, it'll still wait for all Step #1 of other 2 projects and then launch Step #2 of all projects in parallel.

3. Challenge: How can we make ALPHA Project's Step #2 to start as soon as ALPHA project's Step #1 is complete i.e. it won't wait for Step 1 of other 2 projects to complete and could possibly run Step #2 of ALPHA project 1 in parallel with Step N(=1) or N+1 of other projects.

   This assumes all projects are independent of each other and projects don't share contents generated by a given project/their stage/steps in any other project/stage/step.

Depending upon your own requirements, you may want to wait (i.e. don't run Step 2 of all projects until Step 1 of all projects are fully complete) --OR-- you may want to run Step 2 of ALPHA project with let's say - Step 2 of TANGO project while project CHYARLI's step 1 is still in progress.

As the main scope of this post was to get separate dynamically created columns/stages per project (running in parallel within pipeline { ... } section), I think, I got what I was looking for.

NOTE: Go easy with parallel if you want to run concurrent builds of a pipeline. For more info about issues related to running parallel build actions concurrently, see here: Jenkins - java.lang.IllegalArgumentException: Last unit does not have enough valid bits & Gradle error: Task 'null' not found in root project