Jenkins: How To Build multiple top-level projects from a git repository?

Question

I have a Git repository that contains a bunch of top-level maven projects (each sits in their own sub-directory with a pom.xml). Top-level here means that these projects sit in subdirectory directly underneath the repository root. All of these projects should remain in the same Git repository.

repo +--- projectA     +--- pom.xml  +--- projectB     +--- pom.xml 

They can/should be built by independent jenkins jobs. So we have a job for projectA and one for projectB.

Formerly with Subversion I was able to set up a Jenkins job (for each project) that would checkout only the project source and run a Maven build from the pom.xml.

With the Git model (probably the same with all DVCS) this changes and I'm not sure what is best practice. There are a few options that I see and from which none I really like:

1. Each Jenkins job is configured to clone/pull the full Git repo and refers to the /pom.xml for the Maven build. So the job has all the code but builds only a slice of it.
2. Git offers Submodules (http://book.git-scm.com/5_submodules.html) which seem to be a bit tricky to handle (and can break easily)
3. Create a maven parent (aggregator that contains all of the projects) project that triggers each projects build (having a single jenkins job). This pom.xml contains elements for projectA and projectB.

Do you see any more useful approach for this (very typical setup). What is your experience? Any best practices?

Answer 1

I think you're working against the grain here. If these projects are released as a versioned unit, you should indeed create a parent POM. But you should probably only have a single CI job. If you want that build to go quickly, you can configure it to only build modules which have changed since the last build (advanced button in Build section - "Incremental build - only build changed modules"). You can also tell Jenkins to "Execute concurrent builds if necessary" to test multiple commits at once.

But I'm curious why you think you want multiple CI jobs? If you perceive that these two projects have different lifecycles, perhaps they should be versioned seperately and should therefore be in seperate git repositories. Don't conserve git repositories, they are cheap. In fact, the more the merrier in almost every case.

Usually, you want a given pom to produce a single artifact. Aggregator poms are useful for breaking up parts of a larger artifact into submodules, but only if those submodules aren't released on their own.

Answer 2

@recampbell, I have the same problem. We have several interrelated projects in a single Hg repository.

That way we know exactly under which version (hg version number), each other compile. When I say exactly, I mean I can verify it is binary identical.

Using release numbers is too coarse grained. For example today I found a bug that was reproducible only under hg version 3367 but not in hg version 3368. Using multiple hg repos that would have been non reproducible, since there are several commits in each of the projects every day, and therefore it is impossible to determine exactly which version of each individual project was used, only the hg version is the correct one to use (using bisect to find the bug).

It turned out we changed the client database driver version in one of the subprojects and that made our automated dao-generation fail with a NPE. The driver of course if a release version which we don't write, we simply use the one provided by the vendor. It would have taken us several days debugging this, but since we use Jenkins to build every hour, we knew there were between 10 to 20 commits to search, and only 3 or 4 touched the dao-generator, so it was an easy task.

Having mixed versions of every project would have been a pain in the lower rear back, since we would have several different dao-generator-3.1.2.jar which would be a little bit different one from the other (unless you expect us to change the version numbers after every commit, or if there is some configuration in Jenkins/maven/whatever that does this automatically? that would be great...).