Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka connect with mysql custom query

I have done incremental data sync with help of kafka connect. Now i want to achieve same with custom query. But I am getting error.

My config file is

name=mysql-whitelist-timestamp-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector 
tasks.max=1 
connection.url=jdbc:mysql://127.0.0.1:3306/demouser=root&password=root 
query=select name from students3 where marks = 10 
mode=timestamp table.whitelist=students3 
timestamp.column.name=timestamp 
topic.prefix=test-mysql-jdbc-

And getting below error:

ERROR WorkerConnector{id=mysql-whitelist-timestamp-source} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnector:119) org.apache.kafka.connect.errors.ConnectException: query may not be combined with whole-table copying settings.

like image 275
RKP Avatar asked Mar 17 '19 14:03

RKP


People also ask

How connect Kafka connect?

To use Kafka Connect in standalone mode, we need to provide the mandatory parameters: Download a Kafka Connect connector, either from GitHub or Confluent Hub Confluent Hub. Create a configuration file for your connector. Use the connect-standalone.sh CLI to start the connector.


1 Answers

We shouldn't use the tag table.whitelist with the custom query. see the full explanation.

like image 113
Rohit Yadav Avatar answered Nov 15 '22 10:11

Rohit Yadav