Jenkins Pipeline Multiconfiguration Project

Question

Original situation:

I have a job in Jenkins that is running an ant script. I easily managed to test this ant script on more then one software version using a "Multi-configuration project".

This type of project is really cool because it allows me to specify all the versions of the two software that I need (in my case Java and Matlab) an it will run my ant script with all the combinations of my parameters.

Those parameters are then used as string to be concatenated in the definition of the location of the executable to be used by my ant.

example: env.MATLAB_EXE=/usr/local/MATLAB/${MATLAB_VERSION}/bin/matlab

This is working perfectly but now I am migrating this scripts to a pipline version of it.

Pipeline migration:

I managed to implement the same script in a pipeline fashion using the Parametrized pipelines pluin. With this I achieve the point in which I can manually select which version of my software is going to be used if I trigger the build manually and I also found a way to execute this periodically selecting the parameter I want at each run.

This solution seems fairly working however is not really satisfying.

My multi-config project had some feature that this does not:

1. With more then one parameter I can set to interpolate them and execute each combination
2. The executions are clearly separated and in build history/build details is easy to recognize which settings hads been used
3. Just adding a new "possible" value to the parameter is going to spawn the desired executions

Request

So I wonder if there is a better solution to my problem that can satisfy also the point above.

Long story short: is there a way to implement a multi-configuration project in jenkins but using the pipeline technology?

Answer 1

I've seen this and similar questions asked a lot lately, so it seemed that it would be a fun exercise to work this out...

A matrix/multi-config job, visualized in code, would really just be a few nested for loops, one for each axis of parameters.

You could build something fairly simple with some hard coded for loops to loop over a few lists. Or you can get more complicated and do some recursive looping so you don't have to hard code the specific loops.

DISCLAIMER: I do ops much more than I write code. I am also very new to groovy, so this can probably be done more cleanly, and there are probably a lot of groovier things that could be done, but this gets the job done, anyway.

With a little work, this matrixBuilder could be wrapped up in a class so you could pass in a task closure and the axis list and get the task map back. Stick it in a shared library and use it anywhere. It should be pretty easy to add some of the other features from the multiconfiguration jobs, such as filters.

This attempt uses a recursive matrixBuilder function to work through any number of parameter axes and build all the combinations. Then it executes them in parallel (obviously depending on node availability).

/*
    All the config axes are defined here
    Add as many lists of axes in the axisList as you need.
    All combinations will be built
*/
def axisList = [
    ["ubuntu","rhel","windows","osx"],           //agents
    ["jdk6","jdk7","jdk8"],                      //tools
    ["banana","apple","orange","pineapple"]      //fruit
]



def tasks = [:]
def comboBuilder
def comboEntry = []


def task = {
    // builds and returns the task for each combination

    /* Map the entries back to a more readable format
       the index will correspond to the position of this axis in axisList[] */
    def myAgent = it[0]
    def myJdk   = it[1]
    def myFruit = it[2]

    return {
        // This is where the important work happens for each combination
        node(myAgent) {
            println "Executing combination ${it.join('-')}"
            def javaHome = tool myJdk
            println "Node=${env.NODE_NAME}"
            println "Java=${javaHome}"
        }

        //We won't declare a specific agent this part
        node {
            println "fruit=${myFruit}"
        }
    }
}


/*
    This is where the magic happens
    recursively work through the axisList and build all combinations
*/
comboBuilder = { def axes, int level ->
    for ( entry in axes[0] ) {
        comboEntry[level] = entry
        if (axes.size() > 1 ) {
            comboBuilder(axes[1..-1], level + 1)
        }
        else {
            tasks[comboEntry.join("-")] = task(comboEntry.collect())
        }
    }
}

stage ("Setup") {
    node {
        println "Initial Setup"
    }
}

stage ("Setup Combinations") {
    node {
        comboBuilder(axisList, 0)
    }
}

stage ("Multiconfiguration Parallel Tasks") {
    //Run the tasks in parallel
    parallel tasks
}

stage("The End") {
    node {
        echo "That's all folks"
    }
}

You can see a more detailed flow of the job at http://localhost:8080/job/multi-configPipeline/[build]/flowGraphTable/ (available under the Pipeline Steps link on the build page.

EDIT: You can move the stage down into the "task" creation and then see the details of each stage more clearly, but not in a neat matrix like the multi-config job.

...
return {
    // This is where the important work happens for each combination
    stage ("${it.join('-')}--build") {
        node(myAgent) {
            println "Executing combination ${it.join('-')}"
            def javaHome = tool myJdk
            println "Node=${env.NODE_NAME}"
            println "Java=${javaHome}"
        }
        //Node irrelevant for this part
        node {
            println "fruit=${myFruit}"
        }
    }
}
...

Or you could wrap each node with their own stage for even more detail.

As I did this, I noticed a bug in my previous code (fixed above now). I was passing the comboEntry reference to the task. I should have sent a copy, because, while the names of the stages were correct, when it actually executed them, the values were, of course, all the last entry encountered. So I changed it to tasks[comboEntry.join("-")] = task(comboEntry.collect()).

I noticed that you can leave the original stage ("Multiconfiguration Parallel Tasks") {} around the execution of the parallel tasks. Technically now you have nested stages. I'm not sure how Jenkins is supposed to handle that, but it doesn't complain. However, the 'parent' stage timing is not inclusive of the parallel stages timing.

I also noticed is that when a new build starts to run, on the "Stage View" of the job, all the previous builds disappear, presumably because the stage names don't all match up. But after the build finishes running, they all match again and the old builds show up again.

And finally, Blue Ocean doesn't seem to vizualize this the same way. It doesn't recognize the "stages" in the parallel processes, only the enclosing stage (if it is present), or "Parallel" if it isn't. And then only shows the individual parallel processes, not the stages within.