Dynamic Parallel stages in Jenkins Pipeline outside 'script' block

Question

I am trying to construct parallel stages dynamically, as demonstrated here and here. Specifically, I am trying to do this in a function defined outside the scope of the pipeline, e.g.:

pipeline{
  stages{
    stage('CI'){
      steps{
        doDynamicParallelSteps()
      }
    }
  }
}

def doDynamicParallelSteps(){
  tests = [:]
  for (f in findFiles(glob: '**/html/*.html')) {
    tests["${f}"] = {
      node {
        stage("${f}") {
          echo '${f}'
        }
      }
    }
  }
  parallel tests
}

The problem is, it seems this only works when the dynamic parallel generation code is inside a script{} block within the steps{} block of the pipeline (as seen in the first source).

When running something similar to the code snippet above, I see this error in jenkins:

hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.lang.String.call() is applicable for argument types: (java.lang.String, org.jenkinsci.plugins.workflow.cps.CpsClosure2) values: [teststage, org.jenkinsci.plugins.workflow.cps.CpsClosure2@2e1b48b4]
Possible solutions: wait(), any(), trim(), size(), next(), size()
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:153)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
    at WorkflowScript.parallelHandler(WorkflowScript:1383)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Is there any way to have it defined as a function in the way I showed in the initial code snippet, or am I stuck with having a ton of script{} blocks in my pipeline definition?

Answer 1

The declarative pipeline does not allow you to put Groovy code inside steps {} block - it expects a valid Jenkins pipeline step in this place. This is why script {} block got introduced that can be put inside the steps {} block to execute some Groovy code.

If you need flexibility and non-opinionated syntax then you might use scripted pipeline instead. Here you can mix Groovy code with existing pipeline steps with almost no limitations.

Jenkins documentation explains shortly the difference between both approaches and why they exist:

When Jenkins Pipeline was first created, Groovy was selected as the foundation. Jenkins has long shipped with an embedded Groovy engine to provide advanced scripting capabilities for admins and users alike. Additionally, the implementors of Jenkins Pipeline found Groovy to be a solid foundation upon which to build what is now referred to as the "Scripted Pipeline" DSL. [2].

As it is a fully featured programming environment, Scripted Pipeline offers a tremendous amount of flexibility and extensibility to Jenkins users. The Groovy learning-curve isn’t typically desirable for all members of a given team, so Declarative Pipeline was created to offer a simpler and more opinionated syntax for authoring Jenkins Pipeline.

Both are fundamentally the same Pipeline sub-system underneath. They are both durable implementations of "Pipeline as code." They are both able to use steps built into Pipeline or provided by plugins. Both are able to utilize Shared Libraries

Your example in scripted pipeline may look like this:

node {
  stage('CI') {
    doDynamicParallelSteps()
  }
}

def doDynamicParallelSteps(){
  tests = [:]
  for (f in findFiles(glob: '**/html/*.html')) {
    tests["${f}"] = {
      node {
        stage("${f}") {
          echo '${f}'
        }
      }
    }
  }
  parallel tests
}

And declarative pipeline with script {} block in steps would look like this:

pipeline{
  agent any
  stages{
    stage('CI'){
      steps{
        script {
            doDynamicParallelSteps()
        }
      }
    }
  }
}

def doDynamicParallelSteps(){
  tests = [:]
  for (f in findFiles(glob: '**/html/*.html')) {
    tests["${f}"] = {
      node {
        stage("${f}") {
          echo '${f}'
        }
      }
    }
  }
  parallel tests
}