I'm trying to use Apache Flink with AWS kinesis. The document says that I have to build the connector on my own.
Therefore, I build the connector and added the jar file for my project and also, I put the dependency on my pom.xml file.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kinesis_2.11</artifactId>
<version>1.6.1</version>
</dependency>
However, when I tried to build using mvn clean package
I got an error message like this
[INFO] -----------------------< kkaldk:flink-kinesis >-----------------------
[INFO] Building Flink Quickstart Job 0.1
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for org.apache.flink:flink-connector-kinesis_2.11:jar:1.6-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.458 s
[INFO] Finished at: 2018-12-19T17:45:43+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-kinesis: Could not resolve dependencies for project kkaldk:flink-kinesis:jar:0.1: Failure to find org.apache.flink:flink-connector-kinesis_2.11:jar:1.6-SNAPSHOT in https://repository.apache.org/content/repositories/snapshots/ was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced ->
[Help 1]
[ERROR]
Could you help me to fix this?
---------added----------
This is what I've done (which is wrong procedure)
a Kinesis Data Analytics is a fully managed Amazon service that enables you to use an Apache Flink application to process streaming data.
Data streaming sends data to Amazon Kinesis. Open the Amazon Connect console at https://console.aws.amazon.com/connect/ . On the instances page, choose the instance alias. The instance alias is also your instance name, which appears in your Amazon Connect URL.
Performance-wise, Kafka has a clear advantage over Kinesis. Let's not forget that Kafka consistently gets better throughput than Kinesis. Kafka can reach a throughput of 30k messages per second, whereas the throughput of Kinesis is much lower, but still solidly in the thousands.
This is what I've done for solution.
I downloaded 1.6.1 release from here.
unzip the file (the archive will be unzip in .../flink-release-1.6.1/
built it by mvn clean install -Pinclude-kinesis -DskipTests
Then I have flink-connector-kinesis_2.11 version 1.6.1 in my local maven repository
(for me, path was like this): ~/.m2/repository/org/apache/flink/flink-connector-kinesis_2.11
After then, I could use this connector at any project by adding dependency with this code. (I don't have to add the jar file anymore.)
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kinesis_2.11</artifactId>
<version>1.6.1</version>
</dependency>
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With