Accessing Flink Classloader before Stream Start

Question

In my project I would like to access the Flink User Classloader before the stream is executed. I have been instantiating my own Classloader to deserialize classes (doing my best to avoid issues related to multiple classloaders) prior to stream execution.

However the further I am progressing the more issues I am having to write (bad) code to avoid this issue.

This could be solved if I can access the Flink User Classloader and use that, however I don't see a mechanism to do so outside of "RichFunctions" (https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/api/common/functions/RichFunction.html) which require the stream to be running.

Any guidance here would be appreciated

Answer 1

You can use yourself class loader in flink.

Building the graph yourself and submit the class loader with the client.

The code is like follows:

final StandaloneClusterClient client;
try {
    client = new StandaloneClusterClient(configuration);
} catch (final Exception e) {
    throw new RuntimeException("Could not establish a connection to the job manager", e);
}

try {
    ClassLoader classLoader = JobWithJars.buildUserCodeClassLoader(
            Collections.<URL>singletonList(uploadedJarUrl),
            Collections.<URL>emptyList(),
            this.getClass().getClassLoader());
    client.runDetached(jobGraph, classLoader);
} catch (final ProgramInvocationException e) {
    throw new RuntimeException("Cannot execute job due to ProgramInvocationException", e);
}

But I still wonder why you want yourself classloader, it may can be implement by other way.