Linkage failure when running Apache Flink jobs

Question

I have a job developed in Flink 0.9 that is using the graph module (Gelly). The job is running successfully within the IDE (Eclipse) but after exporting it to a JAR using maven (mvn clean install) it fails to execute on the local flink instance with the following error

"The program's entry point class 'myclass' could not be loaded due to a linkage failure"

java.lang.NoClassDefFoundError: org/apache/flink/graph/GraphAlgorithm

Any idea why is this happening and how to solve it?

Answer 1

It looks like the code of flink-gelly did not end up in your jar file. The most obvious reason for this issue is the missing maven dependency in your project's pom file. But I assume the dependency is present, otherwise developing the job in the IDE would be impossible.

Most likely, the jar file has been created by the maven-jar-plugin, which is not including dependencies. Try adding the following fragment to your pom.xml:

    <build>
    <plugins>
        <!-- We use the maven-shade plugin to create a fat jar that contains all dependencies
        except flink and it's transitive dependencies. The resulting fat-jar can be executed
        on a cluster. Change the value of Program-Class if your program entry point changes. -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.3</version>
            <executions>
                <!-- Run shade goal on package phase -->
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <filters>
                            <filter>
                                <artifact>org.apache.flink:*</artifact>
                                <excludes>
                                    <exclude>org/apache/flink/shaded/**</exclude>
                                    <exclude>web-docs/**</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <!-- add Main-Class to manifest file -->
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>YOURMAINCLASS</mainClass>
                            </transformer>
                        </transformers>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>

</build>
<profiles>
    <profile>
        <!-- A profile that does everyting correctly:
        We set the Flink dependencies to provided -->
        <id>build-jar</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <dependencies>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-java</artifactId>
                <version>0.9-SNAPSHOT</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-streaming-core</artifactId>
                <version>0.9-SNAPSHOT</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-clients</artifactId>
                <version>0.9-SNAPSHOT</version>
                <scope>provided</scope>
            </dependency>
        </dependencies>
    </profile>
</profiles>

Now, you can build the jar using mvn clean package -Pbuild-jar. The jar file will now be located in the target/ directory.

You can manually check whether the jar (zip) file contains class files in /org/apache/flink/graph/