Complex and long single-job Jenkins Job Pipeline Builds: There yet?

Question

TL;DR

Does it make sense to have one single Jenking Job Pipeline job for a full long-running deployment / "build" chain in the free Jenkins, given that resumable checkpoints are not available?

Prelude

I'm looking at the current incarnation of Jenkins' the open/free Pipeline Bundle.

I'll note that I think that what you get for free here is quite simply amazing, seeing as most of the stuff (workflow-jobs themselves, stage-view, ...) was only part of the commercial enterprise bundle of CloudBees a year or two back.

Single and Multiple Jobs

It also seems to me that given the stashing abilities and multi-node capabilities of the Jenkins Pipleine, it is -- and indeed is advertised as such -- possible to create a full and complete Build/Test/Bundle/Deploy chain in one single Jenkins job.

This is in contrast to the "historical"(?) Build Pipeline Plugin that let you split "stages" across multiple Jenkins Jobs. Orchestrating these jobs has always been abit of a hassle, not to mention passing artifacts between the jobs and controlling how they run on different nodes.

Repeat/Resume step

Except that what you do get from the Old Build Pipeline Plugin is being able to restart a "stage" (when it fails) -- this works because a "stage" is simply a Jenkins Job Run that you can repeat (and it will also run all the downstream stuff).

When Repeat/Resume is crucial

Our project's C++ Builds have compilation durations of hours, Automated Tests that run longer hours and finally packaging and setup creation that also isn't done in a few minutes.

If I do a "pipleine-build", if the setup creation at the end fails due to setup-creation related problems, I do not want to rebuild the project and re-run the unit tests. I want to fix the setup scripts and redo the setup creation, that would (hopefully) wrap up this run with success. ... Or, if some automated tests fail for, e.g., suspected build environment reasons, I want to fix the build environment and re-run my pipeline from the failed test forward, without doing a multi-hour recompilation first.

Checkpoint

It so happens that the New Jenkins Job Pipeline bundle does have the ability to start from a checkpoint. Except: It's part of the Enterprise Bundle which means it's not free. _{(I'll note that I'd be fine with giving money to CloudBees, but I'm not fine with moving a use-as-many-nodes-as-you-like-and-also-be-able-to-just-do-testruns-wherever-you-like open software to a restricted licensed software where I have to keep track of number of nodes, make sure test environments can even be setup license-wise, etc.)}

The Question: No checkpoint - means still multijob?

So, given that for me the New Jenkins Job Pipeline doesn't have the ability to resume from a failed step, it seems I'm back to square one and I have to split my longer-running stuff into multiple jobs anyway.

This then begs the question whether the Jenkins Job Pipleine is really the best tool for the job -- if I have to orchestrate multiple jobs and I want to streamline and automate this on the creation side, I could just as well use the Job DSL Plugin to maintain all these jobs from a single point -- and then the question is whether "free style" jobs wouldn't do the job just as well.

On the other hand I might be missing something here and there is another good single Job approach for this?

I'll note the Replay option for a Pipeline run looks promising, if a bit crude ...

Answer 1

CloudBees has recently decided to open source the checkpoint plugin, as you can see in the linked issue on the Jenkins Jira that is listed in the comments.

I haven't seen a publicly released version of the plugin yet, though.