Large Java System Dependency Management

Question

We have a large (>500,000 LOC) Java system that depends on 40-50 OSS packages. The system is built with Ant, and dependency management is handled manually at present. I'm investigating Ivy and/or Maven to automate dependencies. We looked at Maven as a build automation system last year and rejected it because it would require totally restructuring our system to match Maven's architecture. Now I'm looking to automate just the dependency management tasks.

I've done some experimentation with Ivy but have run into problems. For example, when I specify ActiveMQ as a dependency, and tell Ivy to use the POMs in the Maven repository for dependency specification, Ivy retrieves a bunch of packages (Jetty, Derby and Geronimo for instance) that I know aren't needed to just use ActiveMQ.

If I set usepoms="false" in ivysettings.xml it fetches only activemq.jar, but that seems to defeat the purpose of Ivy and relegates it to a simple jar-fetcher with manually-built dependency specifications.

There's a bigger issue here, what used to be called "DLL Hell" in Windows. In some cases, two direct first-level dependencies will point to different versions of the same transitive dependency (for instance log4j.jar). Only one log4j.jar can be in the classpath, so dependency resolution involves manually determining which version is compatible with all of its clients in our system.

I guess it all boils down to the quality of each package's dependency specification (the POM). In the case of ActiveMQ, there are no scope declarations, so any reference to ActiveMQ will download all of its dependencies unless we manually exclude the ones we know we don't want.

In the case of log4j, automatic dependency resolution would require that all of log4j's clients (other packages that depend on log4j) validate against all prior versions of log4j and provide a range (or list) of compatible log4j versions in the POM. That's probably too much to ask.

Is this the current state of affairs, or am I missing something?

Answer 1

You're absolutely right in saying that

I guess it all boils down to the quality of each package's dependency specification (the POM).

The only thing I would add is to view the POM, or any other form of metadata, as a starting point. It's quite useful that e.g. ActiveMQ provides all the dependencies for you, but it's up to you to choose if it actually does suit your project.

After all, even taking the log4j version into account, would you have an external dependencies pick the version or choose the version you know works for you?

---

As for how you can choose to tailor dependencies, here's what you can do with Ivy:

Unneeded packages

Ivy retrieves a bunch of packages (Jetty, Derby and Geronimo for instance) that I know aren't needed to just use ActiveMQ.

This usually happens because of poor modularity in the application. Some part of the application needs Jetty for instance, but you end up with this transitive dependency even if you don't use it.

You probably want to look into the ivy exclude mechanism:

<dependency name="A" rev="1.0">   <exclude module="B"/> </dependency> 

Dependency versions

Only one log4j.jar can be in the classpath, so dependency resolution involves manually determining which version is compatible with all of its clients in our system.

Perhaps I'm misreading this, but there is no manual element in Ivy's conflict resolution. There is a list of default conflict managers:

• all: this conflicts manager resolve conflicts by selecting all revisions. Also called NoConflictManager, it does evict any module.
• latest-time: this conflict manager selects only the 'latest' revision, latest being defined as the latest in time. Note that latest in time is costly to compute, so prefer latest-revision if you can.
• latest-revision: this conflict manager selects only the 'latest' revision, latest being defined by a string comparison of revisions.
• latest-compatible: this conflict manager selects the latest version in the conflicts which can result in a compatible set of dependencies. This means that in the end this conflict manager does not allow any conflict (like the strict conflict manager), except that it follows a best effort strategy to try to find a set of compatible modules (according to the version constraints);
• strict: this conflict manager throws an exception (i.e. causes a build failure) whenever a conflict is found.

If needed, you can provide your own conflict manager.

Answer 2

Thats pretty much it. The maven dependency system (which Ivy more or less follows) leaves it up to the individual projects to do a good job of adding the necessary meta data for their dependencies. Most don't.

If you go that route, expect to spend time setting up exclusions.

To the posters recommending OSGi, the OP said that he is not willing to re-architect his build system for Maven, I wouldn't think he would want to re-architect his application to be OSGi compliant. Furthermore, a lot of OSS projects that are OSGi compliant (and there are not as many as you'd hope) have as bad or worse meta data than in Maven