rules and recipes for jenkins build jobs

Question

Currently we're adding groovy pipeline scripts for our CI build in jenkins. Some of the stuff can be parallelized, some depends on the output of some other build step.

parallel "does_not_depend_on_X": { upfront_tests(); },
"depends_on_x": { produce_x(); integration_tests(); publish_test_results()}

Hey! I recognise that! make has solved this with rules and recipes!

all: upfront_tests test_results x

test_results: x
    integration_tests > test_results

x: x_inputs
    produce_x $*

upfront_test_results:
    upfront_tests

Is there a way to achieve this directed-acyclic-graph sorting/parallelizing with jenkins and groovy too?

Answer 1

I must say this is a very interesting question to brainstorm about. AFAIK there is no way to use that syntax. Jenkins supports two ways to define pipelines: scripted pipelines (imperative style using Groovy) and declarative pipelines (a more structured way), although I honestly don't think the latter is really declarative (you still need to define what to do and how to do it), but maybe facilitates building graphical editors because of its simplified syntax. Both are documented here and I don't know of any other way to write them.

One thing to note, however, is that Make (and build tools in general) and Jenkins are very different tools and apply to very different scenarios. Make is a tool to build a program (mainly C or C++ programs), while Jenkins is a full-featured CI automation engine that can be used to implement complex continuous delivery processes starting with an SCM checkout and ending with having an application up and running (usually having passed a set of tests) on an environment. If build tools rely on a directed acyclic graph, a Jenkins pipeline essentially represents a linear workflow across stages. So in a way, the idea of having Jenkins extend Make does not make sense. And even within build tools, they tend to change the way they are being used. For example, Maven and Gradle do not have the concept of a dependency or prerequisite target that Ant or Make have. They still know in which order to run tasks, but the user doesn't have to specify them explicitly.

Another thing is that we have to go back in time to see the idea behind Makefile rules. They are a way to tell Make which files to compile based on which source files have been modified, with the intention of avoiding a full build in a big project. Fundamentally, a dependency in Make is just a source file to check for last modification time. In Jenkins, aside from steps that generate target files from source files, this notion does not exist.

Now you could argue that Jenkins could still use the notion of a dependency to be any prerequisite set of steps to be done before executing the current ones, and not just source files to check when last modified. In other words, we could have a declarative pipeline written as:

pipeline {
   agent any 

   stages {              
       stage(name='Test', depends='Build'){
           steps {
               sh 'make check'
               junit 'reports/**/*.xml' 
           }
       }
       stage('Build') {
           steps { 
               sh 'make' 
           }
       }
       stage(name='Deploy', depends='Test') {
           steps {
               sh 'make publish'
           }
       }
   }
}

Although theoretically this would make the syntax even more declarative (as the order of stages in the code no longer matters), it is practically a useless feature since the order is enough to represent the sequence of stages.