Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to add uid to operator in Flink Table API?

As is highly recommended by the documentation, I want to add uids to my operators in Flink for the purpose of savepointing. My job uses the Table API. I have not found in the documentation how to add uids to operators with a SQL query.

My code looks something like this:

StreamExecutionEnvironment env = ...;
StreamTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env);
Table table = tEnv.sqlQuery("SELECT * FROM mytable GROUP BY TUMBLE(col1, INTERVAL '10' SECOND));
tEnv.writeToSink(table, someSink, qConfig);

If my understanding is correct, the TUMBLE Window is an internal operator state. Therefore, I want to assign it a specific uid to prevent some of the issues that can arise from the autogenerated id. What is the correct way to do this?

I am running Flink v1.6.2

like image 922
Stevenyc091 Avatar asked Sep 18 '25 01:09

Stevenyc091


1 Answers

The Table API does not allow you to set an uid for operators. The problem is that SQL queries might result into different execution plans if one compiles it with a different version. Therefore, it won't help to have the uids set if your plan changes completely. At the moment, it is effectively not possible to provide backwards compatibility for SQL queries.

like image 95
Till Rohrmann Avatar answered Sep 21 '25 13:09

Till Rohrmann