Flink webui when running from IDE

Question

I am trying to see my job in the web ui.

I use createLocalEnvironmentWithWebUI, code is running well in IDE, but impossible to see my job in http://localhost:8081/#/overview

      val conf: Configuration = new Configuration()
      import org.apache.flink.configuration.ConfigConstants
      conf.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true)
      val env =  StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf)
      env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)


      val rides = env.addSource(
        new TaxiRideSource("nycTaxiRides.gz", 1,100))//60, 600))

      val filteredRides = rides
        .filter(r => GeoUtils.isInNYC(r.startLon, r.startLat) && GeoUtils.isInNYC(r.endLon, r.endLat))
        .map(r => (r.passengerCnt, 1))
        .keyBy(_._1)
        .window(TumblingTimeWindows.of(Time.seconds(5)))
        .sum(1)
        .map(r => (r._1.toString+"test", r._2))

      filteredRides.print()
      env.execute("Taxi Ride Cleansing")

Did I need to setup something else?

Answer 1

I was able to start the Flink webui from IntelliJ by adding flink-runtime-web to the dependencies for my project. I did this by adding this to my pom.xml file:

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-runtime-web_2.11</artifactId>
        <version>${flink.version}</version>
    </dependency>

You also then need to create a local execution environment that includes the WebUI:

    Configuration conf = new Configuration();
    env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

Answer 2

Yes, if you want to use WebUI Dashboard, then you need to create an executable jar and then submit this jar to Flink dashboard. I will explain you this step by step

Step 1: Creating the jar from IDE code

• you may need to change your execution environment to

StreamExecutionEnvironment envrionment = StreamExecutionEnvironment.getExecutionEnvironment();

• In case you have multiple jars, then set the main class in Main-Class: variable of Manifest.mf file

• Then create a jar using build artifacts in your IDE

Step 2: Start flink-local cluster which will show you dashboard.

• I will assume that you have not downloaded the Flink binary, you can easily download it here, if you have Macintosh, I will suggest you to use brew install apache-flink which will download the latest stable release which is 1.3.2 currently

• Ok, now you have to go to path where flink is installed and start local cluster

Step# 3 : submitting the job

• submit jar via submit new job option and then run it