Flink's Quickstart doesn't create a proper fat-jar

Question

i'm definitely not an expert in mvn, but after 2 days hacking around, i'm just giving up.

my workflow:

1.     
     mvn archetype:generate                             
    -DarchetypeGroupId=org.apache.flink            
    -DarchetypeArtifactId=flink-quickstart-scala   
    -DarchetypeVersion=0.10.1                       
    -DgroupId=org.apache.flink.quickstart          
    -DartifactId=flink-scala-project               
    -Dversion=0.1                                  
    -Dpackage=org.apache.flink.quickstart          
    -DinteractiveMode=false    

2.    
   cd flink-scala-project 

3.     
  mvn clean package

here is a build log: https://gist.github.com/zavalit/1e78478ebdda827f3454 and when I run

`java -jar target/flink-scala-project-0.1.jar`

I get

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/scala/ExecutionEnvironment$
    at org.apache.flink.quickstart.Job$.main(Job.scala:41)
    at org.apache.flink.quickstart.Job.main(Job.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.scala.ExecutionEnvironment$
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 2 more

Answer 1

The fat jar which you're building is not supposed to be run outside of a cluster environment. Therefore, all Flink related dependencies which run in the cluster environment are excluded from the fat jar.

What you usually do with the generated fat jar is to submit it to a local or remote cluster via bin/flink run -c org.example.MyJob myFatJar.jar. In order to start quickly a local cluster you can run bin/start-local.sh. This will start a local cluster to which you can submit your job jar.

Answer 2

By default, the flink libraries are not included in the fat jar since it would be provided by flink cluster at runtime. To fix that, change the scope of dependencies in pom.xml, from provided to compile:

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java</artifactId>
    <version>${flink.version}</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <scope>compile</scope>
</dependency>

link: Maven Doc