Creating multiple stages with parallel execution on nodes

Question

Our cross platform projects require Jenkins to be executed on a number of different platforms, and appropriate testing and packaging to be done for each. I was able to combine the parallel with node but only in a single stage and got this far (see below)

I want to be able to break it apart into multiple stages. Stages I would like to create are :

• Build: Build the library/project w/ artifacts
• UnitTest: Build and run unit tests
• TestApp: Build and run test applications w/ artifacts
• Input stage: want to upload?
• Upload: Upload artifacts to external server

Would I be doing the following :

1. Copy bunch of objects between stages (using stash)
2. Would need to maintain consistency on node usage. (objects produced on node label should be labelled accordingly be unstashed on the appropriate node label). I would need to label each stash for each node uniquely.

Isn't this a lot more inefficient? I would be artificially copying a lot of data just so that I can create stages.

def checkoutAndBuild(Map args) {
    node("${args.nodeName}") {

        checkout([$class: 'GitSCM', 
                    branches: scm.branches,  
                    doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations, 
                    extensions: scm.extensions + 
                                [[$class: 'SubmoduleOption', 
                                    disableSubmodules: false, 
                                    parentCredentials: false, 
                                    recursiveSubmodules: true, 
                                    reference: '', 
                                    trackingSubmodules: false]] +
                                [[$class: 'CleanCheckout']], 
                    userRemoteConfigs: scm.userRemoteConfigs
                ])

        step([$class: 'CopyArtifact', 
            filter: "AppCommon/*/**, cmake/**/*, core/**/*, thirdparty/prebuilt/${args.prebuiltDir}/**/*, tools/**/*", 
            fingerprintArtifacts: true, 
            projectName: "${args.engineDependency_Job}", 
            selector: [$class: 'SpecificBuildSelector', buildNumber: "${args.engineDependency_BuildNo}"], 
            target: 'engine'])

        dir("build/${args.buildDir}") {
            echo 'Building..'
            if (isUnix()) {
                sh './build.sh Release'
            } else {
                bat 'build.bat Release'
            }
        }

        def extras = args.additionalArtifacts ? ", ${args.additionalArtifacts}" : ""
        archiveArtifacts artifacts: "dist/**/*${extras}", fingerprint: true

        dir("test/build") {
            echo 'Building test App'
            sh "./full.sh ${args.buildDir} Release"
        }
    }
}

pipeline {
    agent none

    stages {
        stage('Info') {
            agent any
            steps {
                echo "Running ${env.JOB_NAME} / ${env.BUILD_ID} on ${env.JENKINS_URL}"
            }
        }

        stage('Build') {
            steps {
                parallel (
                    ios: {
                        checkoutAndBuild nodeName: 'iOS', prebuiltDir: 'ios', buildDir: 'ios', engineDependency_Job: 'engine_iOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.ios)
                    },
                    tvos: {
                        checkoutAndBuild nodeName: 'tvOS', prebuiltDir: 'tvos', buildDir: 'tvos', engineDependency_Job: 'engine_tvOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.tvos)
                    },
                    android: {
                        checkoutAndBuild nodeName: 'Android', prebuiltDir: 'android', buildDir: 'AndroidNative', engineDependency_Job: 'engine_Android_Release', engineDependency_BuildNo: String.valueOf(engineBuild.android), additionalArtifacts: 'src/java/*'
                    })
            }
        }
        stage('Test Build') {
            steps {
                echo 'Testing...'
            }
        }

        stage('Deploy') {
            steps {
                echo 'Deploying...'
            }
        }

    }

    post {
        success {
            slackSend channel: '#builds',
                        color: 'good',
                        message: "${currentBuild.fullDisplayName} succeeded. (<${env.BUILD_URL}|Open>)"

        }
        failure {
            slackSend channel: '#builds',
                        color: 'danger',
                        message: "${currentBuild.fullDisplayName} failed. (<${env.BUILD_URL}|Open>)"
        }
    }
}

Answer 1

Declarative pipeline syntax is not very flexible in your case, unfortunately. Stashing objects between stages would be quite heavy, and it would lead into synchronized stages, where the fastest build target is waiting for slower ones before each stage.

If you don't need to run stages synchronized, I suggest to create one big method which contains all stages of all build targets, those which you want to run parallel. To distinguish between stages, you can for example append node name to each stage label.

def checkoutBuildTestDeploy(Map args) {
    node("${args.nodeName}") {

        stage("Build ${args.nodeName}") {
            checkout([$class: 'GitSCM', ... ])
            // And other build steps ...
        }
        stage("Unit test ${args.nodeName}") {
            // Unit test steps
        }
        stage("Test app ${args.nodeName}") {
            // Test steps
        }
        stage("Deploy ${args.nodeName}") {
            // Input answer and upload
        }
    }
}

pipeline {
    agent none
    stages {
        stage('Info') {
            agent any
            steps {
                echo "Running ${env.JOB_NAME} / ${env.BUILD_ID} on ${env.JENKINS_URL}"
            }
        }

        stage('Run builds parallel') {
            steps {
                parallel (
                    ios: {
                        checkoutBuildTestDeploy nodeName: 'iOS', prebuiltDir: 'ios', buildDir: 'ios', engineDependency_Job: 'engine_iOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.ios)
                    },
                    tvos: {
                        checkoutBuildTestDeploy nodeName: 'tvOS', prebuiltDir: 'tvos', buildDir: 'tvos', engineDependency_Job: 'engine_tvOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.tvos)
                    },
                    android: {
                        checkoutBuildTestDeploy nodeName: 'Android', prebuiltDir: 'android', buildDir: 'AndroidNative', engineDependency_Job: 'engine_Android_Release', engineDependency_BuildNo: String.valueOf(engineBuild.android), additionalArtifacts: 'src/java/*'
                    })
            }
        }
    }
    post {
        success {
            slackSend channel: '#builds',
                      color: 'good',
                      message: "${currentBuild.fullDisplayName} succeeded. (<${env.BUILD_URL}|Open>)"

        }
        failure {
            slackSend channel: '#builds',
                      color: 'danger',
                      message: "${currentBuild.fullDisplayName} failed. (<${env.BUILD_URL}|Open>)"
        }
    }
}