Apache Ivy and local Maven repo - how to handle snapshots built with Maven 3

Question

We currently have a project setup that uses Ivy for dependency management and Ant as the general build tool (although that's probably not relevant here). Additionally we have a bunch of libraries that are build with Maven and which the projects (we have multiple of those) depend on. We know that this situation is far from ideal and we're evaluating ways to improve that but we can't change that as fast as we'd like to. Thus we have to work with what we have at the moment.

Anyhow, here's the problem: it worked with Maven 2 and Ivy but we recently started to switch to Maven 3 for several reasons (one being better conflict resolution) and that combination kind of broke our builds.

First I'll try to describe how our builds worked with Maven 2 and Ivy. After that I'll add where that broke when switching to Maven 3.

Ivy + Maven 2

When developing new versions of our libraries, we're using SNAPSHOT versions, which are installed into the local Maven repo (.m2). Additionally we deploy those snapshots into our Artifactory to be able to share intermediate builds to enable some parallel development.

Our projects then declare dependencies on those snapshots. The corresponding ivysettings.xml looks like this:

<ivysettings>
  <settings defaultResolver="default" />
  <include url="${ivy.default.settings.dir}/ivysettings-local.xml" />
  <resolvers>
    <ibiblio name="public" root="path.to.our.artifactory" m2compatible="true" />

    <filesystem name="local-maven2" m2compatible="true" checkmodified="true" changingPattern=".*SNAPSHOT">      
      <ivy pattern="${user.home}/.m2/repository/[organisation]/[module]/[revision]/[module]-[revision].pom" />
      <artifact pattern="${user.home}/.m2/repository/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
    </filesystem>
    <filesystem name="local" checkmodified="true" changingPattern=".*SNAPSHOT">
      <ivy pattern="${ivy.local.default.root}/${ivy.local.default.ivy.pattern}" />
      <artifact pattern="${ivy.local.default.root}/${ivy.local.default.artifact.pattern}" />
    </filesystem>
    <filesystem name="local2" checkmodified="true" changingPattern=".*SNAPSHOT">
      <ivy pattern="${user.home}/.ivy2/cache/[organisation]/[module]/ivy-[revision].xml" />
      <artifact pattern="${user.home}/.ivy2/cache/[organisation]/[module]/[artifact]-[revision].[ext]" />
    </filesystem>

    <chain name="default" checkmodified="true" changingPattern=".*SNAPSHOT">
      <resolver ref="local" />
      <resolver ref="local-maven2" />
      <resolver ref="public" />
    </chain>
  </resolvers>
  <include url="${ivy.default.settings.dir}/ivysettings-shared.xml" />
</ivysettings>

Due to this setup Ivy should look for newer versions of a snapshot and resolve that correctly. Newer versions are identified by comparing the file date of the corresponding .pom file.

When we build a snapshot with Maven 2 the corresponding .pom will get the file date of the current build and thus that check works and Ivy resolves the correct snapshot versions.

Ivy + Maven 3

The resolution described above breaks when building the snapshots with Maven 3. The reason seems to be that the installed .pom file doesn't get the current timestamp as its file date but keeps the file date of the original pom.xml that is copied. Hence Ivy can't detect whether the snapshot is newer anymore.

It seems as if Maven 3 stores the update timestamps in maven-metadata-local.xml but Ivy doesn't read those. We know that there is the ibiblio resolver (which we are using) but accoring to what we know (e.g. from questions like this) it is meant to work with truly remove repositories and could produce issues when being applied to the local .m2 repo.

We also thought about tracking the file dates of other files, e.g. maven-metadata-local.xml or the actual artifacts, but we're not sure whether that's wise or could break the build in other situations.

So how should/could we solve that?

TL;DR

What is the standard/suggested way to handle Ivy dependencies on snapshots that are built with Maven 3 and deployed in the local .m2 repo?

UPDATE 2016-07-26

Here are 2 things I tried to solve the issue but I'm not sure whether there would be any side effects I didn't think of. I'd be grateful if someone could shed some light on this:

1. Use the filesystem resolver for the local .m2 repository but in conjection with a cache which has useOrigin="true" (and optionally a low defaultTTL). That way it seems as if only the xml files are stored in the .ivy2 cache and the artifacts are referenced in the .m2 repository, i.e. they aren't copied.
   
   This seems to work but I'm not sure whether it would if the first lookup would download the snapshot from out shared snapshot repository (Artifactory) and those would be updated by a local build later. In that case we'd probably end up with the remote version of the artifact being cached in .ivy2 and a newer version being installed into .m2 with the .pom file dates being the same in both cases.
2. Use the ibiblio resolver with root="file://${user.home}/.m2/repository/" which seems to be able to resolve most artifacts (exception see below) but still seems to fail when reading the metadata, i.e. it doesn't update the cache with the newer version which is to be found in the .m2 repo.
   
   While being able to resolve most artifacts in the .m2 repo , I seem to get resolution errors for a few "urls" like file://{user.home}/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar while the artifacts exist, i.e. I copied the path directly from Windows Explorer and it is "${user.home}\.m2\repository\javax\enterprise\cdi-api\1.0\cdi-api-1.0.jar".

Answer 1

... We know that this situation is far from ideal and we're evaluating ways to improve that but we can ...

Perfectly normal in my experience. Many large companies have lots of projects built with a mixture of ANT, Maven and increasingly Gradle.

What follows are a couple of recommendations I would make

Use a repository manager

The best way to integrate these different technologies is to run an intermediate repository to hold all build outputs. This will effectively decouple your build step from how the binary artefacts are later consumed.

The recommended repository technology a Maven repository (Java standard at this point) and you have a choice of open source technologies available to host your repository:

• Nexus
• Artifactory
• Archiva

While running an extra server might appear to be an overhead it pays back in spades as the number of dependent projects increase (having a single large shared filesystem does not scale).

In any case you need to have a reference copy of your release binaries anyway. These Maven repository managers allow you to control the 3rd party dependencies your projects consume from and helpfully cache files which would actually decrease your build times and simplify shared dependency management.

Finally how does one configure an ivy build that uses a Maven repository? As follows using the ibiblio resolver (and you'll discover that all modern Java build tools have similar support for a local Maven repository)

<ivysettings>
    <settings defaultResolver="myrepo"/>
    <resolvers>
        <ibiblio name="myrepo" m2compatible="true" root="http://myrepo.com/path/to/repo"/>
    </resolvers>
</ivysettings>

Beware of snapshot releases

Snaphot releases are a concept introduced by Maven, which is an opinionated build framework. In my opinion one should be very deliberate when using them:

• Snapshots are for Dev, Release Candidates are for QA

As you've discovered snapshots are constantly changing and cannot be relied upon when sharing with a group who expect an unchanging arefect for testing (like QA).

My thinking on this has been shaped by the following posting that discusses the Maven approach to release management:

• http://www.slideshare.net/wakaleo/continuous-deliverywithmaven

Lastly, while I'd recommend limiting the use of snapshots to closely collaborating teams, ivy does support them, there are just some well known limitations to their use. Because they're a Maven construct they're not 100% supported by other build technologies. This is where a Repository manager really helps as they're capable of regenerating the correct metadata to support snapshot releases:

• What's wrong with this Ivy changingPattern / SNAPSHOT configuration?
• Publishing Ivy SNAPSHOTS with Maven metadata

Hope this helps.