Best way to automatically check out and compile Eclipse projects with Ant in Hudson or another CI tool?

Question

We have several products which have a lot of shared code and which must be maintained several versions back.

To handle this we use a lot of Eclipse projects, some contain library jars, and some contain shared source code (in several projects to avoid getting a giant heap with numerous dependencies while being able to compile everything from scratch to ensure that source and binaries are consistent). We manage those with projectSet.psf's as these can directly pull all projects out from CVS and leave a fully prepared workspace. We do not do ant builds directly or use maven.

We now want to be able to put all these projects and their various versions in a Continous Integration tool - I like Hudson but this is just a matter of taste - which essentially means that we need to get an automatic way to check out the projects to a fresh workspace, and compile the source folders as described in the project-files in each project. Hudson does not provide such an approach to build a project, so I have been considering what the best way to approach this would be.

Ideas have been

• Find or write an ant plugin/converter that understands projectSet.psf's and map to cvs-checkout and compile tags.
• Create the build.xml files from within Eclipse and use those. I tried this, and found the result to be verbose and with absolute locations which is not good with automatic tools putting files where they want to.
• Write a Hudson plugin which understands projectSet.psf's to derive a configuration and build it.
• Just bite the bullet and manually create and update the CI configuration whenever stuff breaks - I don't like this :)

I'd really like to hear about other peoples experiences so I can decide how to approach this.

---

Edit: Another option might be using a CI which knows better about Eclipse projects and/or project sets. We are not religious - this is just a matter of getting stuff running without having to do everything ourselves. Would Cruise Control be a better option perhaps? Others?

---

Edit: Found that ant4eclipse has a "Team Project Set" facility. http://ant4eclipse.sourceforge.net/

---

Edit: Used the ant4eclipse and ant-contrib ant extensions to build a complete workspace as a sjgned runnable jar file similar to the Runnable Jar facility in Eclipse 3.5M6. I am still depending on Eclipse to create the initial empty workspace, and extract the ProjectSet, so that is the next hurdle.

---

Edit: Ended up with a dual configuration, namely that Hudson extracts the same set of modules as listed in the ProjectSet.pdf file from CVS (which needs to have the same tag) causing them to be located next to each other. Then ant4eclipse works well with the projectSet.psf file embedded in the main module. Caveat: Module list in Hudson must be manually updated, and it appears that a manual workspace cleanup is needed afterwards to let Hudson "discover" that there is more projects now than earlier. This has now worked well for us for a couple of months, but it was quite tedious to get everything working inside the ant file.

---

Edit: The "Use Team Projects" with ant4eclipse and a Ctrl-A, Ctrl-C in Project Panel with a Ctrl-V in the CVS projects in Hudson has turned out to work well enough for us to live with (for mature projects this is very rarely changed). I am awaiting the release of ant4eclipse 1.0 - http://www.ant4eclipse.org/, currently in milestone 2 - to see how much homegrown functionality can be replaced with ant4eclipse things.

Edit: ant4eclipse is as of 20100609 in M4 so the schedule at http://www.ant4eclipse.org/node?page=1 is slipping somewhat.

---

Edit: My conclusion after using our ant4eclipse approach for a longer period is that the build script get very gnarly and is hard to maintain. Also the Team ProjectSet facility (which ant4eclipse use to locate the projects) which works well for CVS based repositories, but not after we migrated to git (which is a big thing in itself). New projects will most likely be based on maven, as this has good support in Jenkins.

Answer 1

I'm not completely sure I understand the problem, but it sounds like the root issue is that you have many projects, some of which are dependent on others. Some of the projects that are closer to the "leaf" of the dependency tree need to be able to use "stable" (or previously "released") versions of the more "core" projects.

I solve exactly this problem using Hudson, ant, and ivy. I follow a pattern demonstrated by Clark in Pragmatic Project Automation (he doesn't demonstrate the dependency problems and solutions, and he uses CruiseControl rather than hudson.)

I have a hand-written ant build file (we call it "cc-build.xml", because of our CruiseControl roots.) This file is responsible for refreshing the working space for the project from the CM repository and labeling the contents for future reference. It then hands off control to another hand-written ant build file (build.xml) that is provided by each project's developers. This project is responsible for the traditional build steps (compile, packaging, etc.) It is required to spit out the installable artifacts, unit test reports, etc, to the Hudson artifacts directory. It is my experience that automatically generated build files (by Eclipse or other similar IDE's) will never get close to getting this sufficiently robust for use in a CI scenario.

Additionally, it uses ivy to resolve its own dependencies. Ivy supports precisely-specified dependency versions (e.g. "use version 1.1") and it supports "fuzzy versions" (e.g. "use version 1.1+" or "use the latest version in integration status.") Our projects typically start out specifying a very "fuzzy" version for internal projects under ongoing development, and as they get close to a release point, they "freeze" the dependency version so that stuff stops moving underneath them.

The non-leaf projects (projects that are dependents for other projects) also use ivy to publish their artifacts to our internal ivy repository. That repository keeps all past builds of the dependents, so that any project can always depend on any other previous version.

Lastly, each project in Hudson is configured to have a build trigger that causes a rebuild when any of its dependent projects successfully build. This causes them to get built again with the (possibly) new ivy dependent version.

It is worth noting that once you get this up and running, consistent automated "labeling" or "tagging" of an automated build's inputs is going to be critical for you - otherwise troubleshooting post-build problems is going to result in having to untangle a hornet's nest to find the original source.

Getting all of this setup for our environment took quite a bit of effort (primarily in setting up the ivy repository and ant build files,) but it has paid for itself many times over in saved headaches in manually managing the dependencies and decreased troubleshooting effort.