Making the Storm JARs compile-time only in a Gradle project

Question

I am trying to build a Gradle project which contains a Storm project. In order to run this project on Storm, I have to first create a JAR file and let Storm run my topology, e.g.

storm jar myJarFile.jar com.mypackage.MyStormMainClass

I am running into problems because Gradle, by default, is including the Storm dependencies both at compile time and runtime. This causes the following exception:

Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.

The Exception given is actually helpful, and tips us off to the root cause of the problem. The solution is to include the Storm dependencies when compiling with Gradle, but not when generating the final JAR file.

Does anyone know how to resolve this? The other posts on StackOverflow did not solve the problem. If you paste code please ensure that it actually runs.

Thanks!

Answer 1

Below I am quoting an answer of mine to a similar thread on the storm-user mailing list.

In my case I opted to use the fatJar plugin for Gradle for that purpose.

buildscript {
    repositories {
        mavenCentral()
        mavenRepo url: "http://clojars.org/repo"
    }
    dependencies {
        // see https://github.com/musketyr/gradle-fatjar-plugin
        classpath 'eu.appsatori:gradle-fatjar-plugin:0.2-rc1'
    }
}

// When using this plugin you can build a fat jar / uberjar with 'gradle fatJar'
apply plugin: 'fatjar'

dependencies {
    compile 'storm:storm:0.8.2', {
        ext {
            fatJarExclude = true
        }
    }
}

You build the fat jar via:

$ gradle clean fatJar

Best, Michael

PS: For what it's worth I also blogged about how to solve this problem as described above at Running a Multi-Node Storm Cluster. That post contains additional pieces of information and gotchas that may or may not be helpful to you.