Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to build and use flink-connector-kinesis?

I'm trying to use Apache Flink with AWS kinesis. The document says that I have to build the connector on my own.

Therefore, I build the connector and added the jar file for my project and also, I put the dependency on my pom.xml file.

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kinesis_2.11</artifactId>
    <version>1.6.1</version> 
</dependency>

However, when I tried to build using mvn clean package I got an error message like this

[INFO] -----------------------< kkaldk:flink-kinesis >-----------------------
[INFO] Building Flink Quickstart Job 0.1
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for org.apache.flink:flink-connector-kinesis_2.11:jar:1.6-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.458 s
[INFO] Finished at: 2018-12-19T17:45:43+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-kinesis: Could not resolve dependencies for project kkaldk:flink-kinesis:jar:0.1: Failure to find org.apache.flink:flink-connector-kinesis_2.11:jar:1.6-SNAPSHOT in https://repository.apache.org/content/repositories/snapshots/ was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -> 
[Help 1]
[ERROR] 

Could you help me to fix this?

---------added----------

This is what I've done (which is wrong procedure)

  1. started project following this document
  2. build connector from flink master source (which is 1.8-snapshot)
  3. add dependency with 1.6.1 version.

like image 672
Bumhwan Kim Avatar asked Dec 19 '18 09:12

Bumhwan Kim


People also ask

Does Kinesis use Flink?

a Kinesis Data Analytics is a fully managed Amazon service that enables you to use an Apache Flink application to process streaming data.

How do I connect to AWS Kinesis?

Data streaming sends data to Amazon Kinesis. Open the Amazon Connect console at https://console.aws.amazon.com/connect/ . On the instances page, choose the instance alias. The instance alias is also your instance name, which appears in your Amazon Connect URL.

Is Kinesis better than Kafka?

Performance-wise, Kafka has a clear advantage over Kinesis. Let's not forget that Kafka consistently gets better throughput than Kinesis. Kafka can reach a throughput of 30k messages per second, whereas the throughput of Kinesis is much lower, but still solidly in the thousands.


1 Answers

This is what I've done for solution.

  1. I downloaded 1.6.1 release from here.

  2. unzip the file (the archive will be unzip in .../flink-release-1.6.1/

  3. built it by mvn clean install -Pinclude-kinesis -DskipTests

Then I have flink-connector-kinesis_2.11 version 1.6.1 in my local maven repository

(for me, path was like this): ~/.m2/repository/org/apache/flink/flink-connector-kinesis_2.11

After then, I could use this connector at any project by adding dependency with this code. (I don't have to add the jar file anymore.)

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kinesis_2.11</artifactId>
    <version>1.6.1</version> 
</dependency>
like image 189
Bumhwan Kim Avatar answered Oct 18 '22 11:10

Bumhwan Kim