I'm guessing this is more of a general Java/Eclipse question, but I'm not a Java guy and this isn't clicking for me. Stack trace at the end.
https://github.com/apache/beam/blob/9d9ffa5f1a3a9f280dfafae15944764a568515ef/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
Above is the file where the error is being thrown and it's pretty straightforward - are we using a URLClassLoader
or not? Not being a Java guy, I don't understand how to switch to a URLClassLoader
and googling this makes me feel like it's not something I need to be concerned with.
"Unable to use ClassLoader to detect classpath elements"
Makes me feel like I added a classpath incorrectly and Eclipse is doing something weird.
None of the Google dataflow/apache beam documentation mention anything about a URLClassLoader
and now I can't even get their example project to run. I'm pretty sure I did something, but I'm not sure what.
Any help would be appreciated.
Jan 16, 2018 8:03:48 PM org.apache.beam.runners.dataflow.DataflowRunner
detectClassPathResourcesToStage
SEVERE: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@3b764bce, only URLClassLoaders are supported.
Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
at twitchInsights.MainPipeline.main(MainPipeline.java:59)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more
Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@3b764bce, only URLClassLoaders are supported.
at org.apache.beam.runners.dataflow.DataflowRunner.detectClassPathResourcesToStage(DataflowRunner.java:1365)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:244)
... 9 more
Full Exception:
Thread [main] (Suspended (exception IllegalArgumentException))
DataflowRunner.detectClassPathResourcesToStage(ClassLoader) line: 1365
DataflowRunner.fromOptions(PipelineOptions) line: 244
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 564
InstanceBuilder<T>.buildFromMethod(Class<?>[]) line: 222
InstanceBuilder<T>.build() line: 162
PipelineRunner<ResultT>.fromOptions(PipelineOptions) line: 52
Pipeline.create(PipelineOptions) line: 142
MainPipeline.main(String[]) line: 60
A workaround that works for me in Java 9 is defining filesToStage programmatically based on the classpath.
Her is an example how you can generate the file set from the class path:
dataflowPipelineOptions.setFilesToStage(
Arrays.
asList(System.getProperty("java.class.path").split(File.pathSeparator)).
stream().
map(entry -> new File(entry).toString()).
collect(Collectors.toList())
)
I think this is a Java 9 specific issue. Beam does not officially support Java 9 yet - see https://issues.apache.org/jira/browse/BEAM-2530 for current progress.
You can try working around this specific issue by specifying the classpath elements via --filesToStage
instead (this will bypass the autodetection based on current classloader - which is what's failing). However you may hit other issues.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With