Ideas to implement dynamic parallel build using jenkins pipeline plugin

Question

I have a requirement to run a set of tasks for a build in parallel, The tasks for the build are dynamic it may change. I need some help in the implementation of that below are the details of it.

I tasks details for a build will be generated dynamically in an xml which will have information of which tasks has to be executed in parallel/serial

example:

say there is a build A.

Which had below task and the order of execution , first task 1 has to be executed next task2 and task3 will be executed in parallel and next is task 4

task1
task2,task3
task4

These details will be in an xml dynamically generated , how can i parse that xml and schedule task accordingly using pipeline plugin. I need some idea to start of with.

Answer 1

You can use Groovy to read the file from the workspace (readFile) and then generate the map containing the different closures, similar to the following:

parallel(
  task2: {
    node {
      unstash('my-workspace')
      sh('...')
    }
  },
  task3: {
    node {
      unstash('my-workspace')
      sh('...')
    }
  }
}

In order to generate such data structure, you simply iterate over the task data read using XML parsing in Groovy over the file contents you read previously.

By occasion, I gave a talk about pipelines yesterday and included very similar example (presentation, slide 34ff.). In contrast, I read the list of "tasks" from another command output. The complete code can be found here (I avoid pasting all of this here and instead refer to this off-site resource).

The kind of magic bit is the following:

def parallelConverge(ArrayList<String> instanceNames) {
    def parallelNodes = [:]

    for (int i = 0; i < instanceNames.size(); i++) {
        def instanceName = instanceNames.get(i)
        parallelNodes[instanceName] = this.getNodeForInstance(instanceName)
    }

    parallel parallelNodes
}

def Closure getNodeForInstance(String instanceName) {
    return {
        // this node (one per instance) is later executed in parallel
        node {
            // restore workspace
            unstash('my-workspace')

            sh('kitchen test --destroy always ' + instanceName)
        }
    }
}