Failed to construct instance from factory method DataflowRunner#fromOptions in beamSql, apache beam

Question

I'm specifying dataflow runner in my beamSql program below :

DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
    options.setStagingLocation("gs://gcpbucket/staging");
    options.setTempLocation("gs://gcpbucket/tmp");
    options.setProject("beta-19xxxx");
    options.setRunner(DataflowRunner.class);
    Pipeline p = Pipeline.create(options);

But I'm getting below exception :

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at my.proj.StarterPipeline.main(StarterPipeline.java:34)Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@782830e, only URLClassLoaders are supported.
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:43)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:262)

can anyone help me out understand what exactly is the issue ?

Answer 1

Downgrading java 9 to java 8 did the trick.

Answer 2

This problem can happen when using Beam SDK with JVM > 8.

Support for JVM 11 (or 9) is only experimental, starting from Beam SDK 2.12 and at least up to Beam SDK 2.16. This answer provides a bit more context: https://stackoverflow.com/a/57710742/1046584

Regarding this specific stack trace, it seems related to this issue: https://issues.apache.org/jira/browse/BEAM-3718

Answer 3

For me the fix was different. In my pom.xml, below dependency was set to 2.20 while apache beam was set to 2.19

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
  <version>2.20</version>
</dependency>

After changing it to same as beam version, build was working fine.

Net Net, just make sure all the beam related sdks are set to same version. Use variable for the same, like ${beamVersion} so you don't end up facing strange errors.