Can Jenkins load shared libraries without recloning them at each execution?

Question

I find Jenkins Shared (Groovy) Libraries name bit confusing and ironic after I discovered that Jenkins would clone the shared-library repository for each build, yes cloning the same code for each execution.

This is against the concept of a shared-library: as these are pieces of code that are loaded by multiple consumers. Imagine how it would be if the operating system would make a copy of any loaded library when you try to load it... (yep thousands per day)

Is there a way to avoid this serious overload on resources?

Reference:

• https://jenkins.io/doc/pipeline/steps/workflow-cps-global-lib/
• https://jenkins.io/blog/2017/06/27/speaker-blog-SAS-jenkins-world/

Answer 1

I think there is currently no way to avoid that, but I agree that this would be a huge benefit.

Jenkins should cache the shared libraries and only pull the latest branch referenced in the Jenkinsfile.

Another great benefit of locally cached shared libraries, apart from saving the constant cloning overhead, would be independence towards your repository. Jenkins should be able to use the last shared lib version it cached - even if the repository is down. Currently, if you have pipelines that don't only build but also deploy, then your deployment process is depending on the availability of your shared library repository - but ideally one can deploy already built packages even in case the git repository is down.

Answer 2

This feature has now been made available, via https://issues.jenkins.io/browse/JENKINS-38992

The versions 2.21 (and above) for plugin https://github.com/jenkinsci/workflow-cps-global-lib-plugin/releases has that fix. It can be found in the latest stable and LTS versions as of late 2021.

When declaring your Global Pipeline Libraries in the global Settings, you have new fields to enable caching per libraries. Also, you can designate versions (i.e. branches) that won't be cached.

If you use JCasC, you can use the syntax:

          globalLibraries:
            libraries:
              - name: "shared-lib"
                cachingConfiguration:
                  excludedVersionsStr: "master develop"
                  refreshTimeMinutes: 0
                retriever:
                  ...