Can I run a Jenkins Pipeline step or stage on individual schedules?

Question

I am relatively new to Jenkins pipelines and wondered if it's possible to port my current build configuration to a Jenkinsfile.

At the moment for every project has a CI Job that builds as soon as someone checks in. This is just a build check and does no tests or static code analysis. A second Job does the full build, including long running integration tests and a sonarqube analysis. The second one is scheduled to run once a day (the nightly) no matter if changes in this particular project happened or not.

Now if i'd like to port that configuration to Jenkins Pipeline scripts i obviously can configure two Jobs with individual scripts but the much more interesting part would be to wrap it into a Jenkinsfile so i can make use of the Multibranch Pipeline Job Type and do not have to repeat any configurations outside the SCM.

The question is, if it is possible to defer build steps (and abort the ones from previous builds that are no longer needed) to another Schedule - e.g. run the integration tests only on a nightly base? How do others adopt to the problem having fast build checks vs. long running tasks that might not be useful on every check-in?

I also haven't seen a possibility to select another name for the Jenkinsfile (in the multibranch Pipeline Job Type) so one could have one file for the CI job and another for the nightly. A condition inside the script evaluating the job name might be possible but doesn't look "right" for me and it does not solve to problem forcing a build every evening.

Thank you for any ideas Daniel

Answer 1

Well now that I had more time to investigate there seem to several ways to go.

guard time intensive stages with a timed out stage

The following code shows only the way I think it could work. I have not done it this way, so be prepared for some tweaking:

node {
    stage('build'){
        // sh 'mvn install'
        // most likely you'll need some stashing here to restore
        // the current state in the next node
    }
}
// it' essential to leave the node or you'll block the executor
stage('acknowledge'){
    // https://jenkins.io/doc/pipeline/steps/pipeline-milestone-step/#milestone-the-milestone-step-forces-all-builds-to-go-through-in-order
    milestone // mark before entering
    try {
        // here one needs to implement the magic to calculate how much time is left to the regular nightly
        // remember to slightly decrease the time for every build, so that the last will be the first to continue
        def timeToNightly = calculateTimeToNightly() 

        // https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-timeout-code-enforce-time-limit
        timeout(time: timeToNightly, unit: 'MILLISECONDS'){
            // having a input leaves the option to continue a build before the timeout is reached
            input 'continue with integration tests?'
        }
    } catch(e) {
        // so it's time for the daily hardwork
    }
    milestone // abort all older builds
}
node {
    stage('test: integration'){
        // possibly unstash previous build files
        // sh 'mvn verify'
    }
}

enable/disable stages based on the causer

One may iterate over the causer and deactivate functions in stages based on SCM changes, user or timer triggered builds. https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/get-build-cause/getBuildCause.groovy

So you'll need to add a TimerTrigger to the build that suites your needs, e.g. only every 2nd thursday of the month (wich may be difficult with the variant from above). https://jenkins.io/doc/pipeline/steps/workflow-multibranch/#properties-set-job-properties

properties([
    pipelineTriggers([
        // this is a timer trigger you may adjust the cron expression to your needs
        cron('@daily')
    ])
])

node {
    stage('build'){
        // sh 'mvn install'
    }
}

// here you'll have to go over the build causer and adjust the return value accordingly
if (isTriggeredByTimer()) {
    node {
        stage('test: integration'){
            // sh 'mvn verify'
        }
    }
} else {
    // repeat the stages here so they won't disappear in the job view
    // doing this outside a node is better than to block a build slot first
    stage('test: integration'){
        echo 'Skipped testing for this build - it not the time yet.'
    }
}

I tweaked this even more to provide Job Parameters so that users may make their own decisions (do the tests or only static code analysis or both or none of them...).

Hope that helps.

As for me, now that I see all of the potential the structured pipeline offers (declarative is easier to adapt, but not as powerful) to tweak the build, abort older ones, limit executions with lockable resources and continue only with the latest (milestones), I may try to just keep executing every step and let only the latest win.