Java Project Layout Best Practices for Ant-based Builds

Question

I'm a little shocked that (if?) this precise questions hasn't been asked, but 15 minutes of scanning SO didn't turn up an exact match. (If I'm wrong, point me the right way and vote to close...)

Question 1:

What are the best practices for laying out Java projects under an Ant build system? For our purposes, we have the following context (perhaps most of which is irrelevant):

• Most developers are using Eclipse (not all)
• Project is maintained in subversion
• Project builds have recently migrated to Hudson, in which we want to use the release plugin to manage releases, and some custom scripts to handle automated deployment
• This project is a "conventional" application, a sort of "production prototype" with a very limited pool of users, but they are at remote sites with airgap separation, so delivering versioned, traceable artifacts for easy installation, manual data collection/recovery, and remote diagnosis is important.
• Some dependencies (JARs) are included in the SVN repo; others may be fetched via the ant script at build time, if necessary. Nothing fancy like Ivy yet (and please don't tell me to switch to Maven3... I know, and we'll do so if/when the appropriate time comes.)
• Build includes JUnit, FindBugs, CheckStyle, PMD, JavaDoc, a bit of custom documentation generation
• Two or three primary JAR artifacts (a main application artifact plus a couple of minimal API JARs for inclusion in a few coupled applications)
• Desire to distribute the following distribution artifacts:
  • a "Full" source+bin tarball with all dependencies resolved, jars and JavaDoc prebuilt
  • a bin tarball, with just the docs and JavaDoc, jars, and ancillary wrapper scripts etc
  • a "Partner" source+bin, which has the "shared" source that partner developers are likely to look at, and associated testcases

Current structure looks like this

project-root/
project-root/source                 
project-root/source/java             // main application (depends on -icd)
project-root/source/java-icd         // distributable interface code
project-root/source/test             // JUnit test sources
project-root/etc                     // config/data stuff
project-root/doc                     // pre-formatted docs (release notes, howtos, etc)
project-root/lib                     // where SVN-managed or Ant-retrieved Jars go
project-root/bin                     // scripts, etc...

At build time, it expands to include:

build/classes                        // Compiled classes
build/classes-icd 
build/classes-test
build/javadoc
build/javadoc-icd                    
build/lib                            // Compiled JAR artifacts
build/reports                        // PMD, FindBugs, JUnit, etc... output goes here
build/dist                           // tarballs, zipfiles, doc.jar/src.jar type things, etc..
build/java                           // Autogenerated .java products
build/build.properties               // build and release numbering, etc...

Question 2:

How can I maintain strict separation in the development tree between revision-controlled items and build-time artifacts WHILE producing a coherent distribution as above AND allowing me to treat a development tree as a operational/distribution during development and testing? In particular, I'm loathe to have my <jar> task drop .jar files in the top-level lib directory -- that directory in the developers' trees is inviolable SVN territory. But distributing something for public use with build/lib/*.jar is a confusing annoyance. The same is true of documentation and other built artifacts that we want to appear in a consistent place in the distribution, but don't want to have developers and users use completely different directory structures.

Having all the generated products in a separate build/ directory is very nice for development-time, but it's an annoying artifact to distribute. For distribution purposes I'd rather have all the JARs sitting in a single lib location, in fact, a structure like the below makes the most sense. Currently, we build this structure on the fly with ant dist by doing some intricate path manipulations as .tar.gz and .zip artifacts are built.

What I think the dist should look like:

project-root/
project-root/source                  // present in only some dists 
project-root/etc                     // same as in development tree
project-root/doc                     // same as in development tree
project-root/doc/javadoc             // from build 
project-root/lib                     // all dependency and built JAR files
project-root/bin                     // same as in development tree
build.properties               // build and release numbering, etc...

This question is narrowly about the "how do I maintain clean development and distribution project layouts?" as I asked above; but also to collect info about Java/Ant project layouts in general, and critiques of our particular approach. (Yes, if you think it should be a Community Wiki I'll make it so...)

Answer 1

My one suggestion would be that the directory tree you distribute should not be the one in CVS. Have a script which puts together a dist directory under build, then zips that up. That script can combine source-controlled and derived files to its heart's content. It can also do things like scrub out SVN directories, which you don't want to distribute. If you want to be able to treat development and distributed trees in the same way, simply ensure that the layout of dist is the same as the layout of the development project - the easiest way to do that would be to copy everything except the build subdirectory (and CVS directories, and perhaps things like the Eclipse .project and .classpath).

I suspect you won't like this suggestion. It may be that you are attached to the idea that the distributed file is simply a portable version of your development environment - but i think it's the case that it isn't, it can never be, and it doesn't need to be. If you can accept that idea, you might find my suggestion agreeable.

EDIT: I thought about this a bit more, and looked at some of the scripts i use. I think what i'd do in this situation is to build a separate tree even in development; point the execution environment at project-root/build/app (or perhaps project-root/build if you can) rather than project-root, and then symlink (or copy if you don't have symlinks) all the necessaries (whether static, from in the project root, or derived, from in build) into that. Building a distribution may then be as simple as zipping up that tree (with a tool that resolves symlinks, of course). The nice thing about this is it allows the structure of the executed tree to be very clean - it won't contain source directories, IDE files, build scripts, or other supporting files from inside the project, etc. If you're using Subversion, it will still contain .svn directories inside anything symlinked from the static areas; if you were using Mercurial, it wouldn't contain any .hg stuff.

Answer 2

Layout-wise, we use something which has evolved into something very close to a Maven layout (see here). This is a very functional layout which has been used by a lot of people. And, if you want to switch to Maven later, you're all set. We have a couple of variations, the most important of which is that we separate automated unit- and integration-tests.

In terms of mingling sources and build artefacts - I would certainly recommend against it. As you've seen, it messes with IDE indexing and version control and generally makes life difficult.

As far as I can tell you either have to accept this mingling, or copy your dependencies as part of the build and treat the output as a separate project - perhaps constantly open in another IDE window if you need it. The idea of mixing your work 'as a user' versus 'as a producer' of your release package sounds like it would be confusing, anyway.

Answer 3

http://ant.apache.org/ant_in_anger.html

The project contains sub directories

• bin common binaries, scripts - put this on the path.
• build This is the tree for building; Ant creates it and can empty it in the 'clean' project.
• dist Distribution outputs go in here; the directory is created in Ant and clean empties it out
• doc Hand crafted documentation
• lib Imported Java libraries go in to this directory
• src source goes in under this tree in a hierarchy which matches the package names.