Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

assertion failed: unsafe symbol DeveloperApi in runtime reflection universe

Tags:

java

scala

apache-spark

I'm running the following lines in a Java 8 program

SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("testJob");
sparkConf.setMaster("spark://blahblah:7077");
SparkSession sparkSession = SparkSession.builder().config(sparkConf).getOrCreate();
SQLContext sqlContext = new SQLContext(sparkSession);
Dataset<Row> ds = sqlContext.sql("SHOW TABLES");

and I get the quite mysterious exception (running the sqlContext.sql("SHOW TABLES") line)

java.lang.RuntimeException: error reading Scala signature of
    org.apache.spark.sql.package: assertion failed: unsafe symbol 
    DeveloperApi (child of package annotation) in runtime reflection universe

(The full stacktrace is reported at the end of this question).

Can anyone give me a hint on how to solve this? What does it mean?

[Gogo shell] INFO org.apache.spark.util.Utils - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51077.
[Gogo shell] INFO org.apache.spark.network.netty.NettyBlockTransferService - Server created on 172.29.30.215:51077
[Gogo shell] INFO org.apache.spark.storage.BlockManagerMaster - Registering BlockManager BlockManagerId(driver, 172.29.30.215, 51077)
[dispatcher-event-loop-0] INFO org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint - Executor updated: app-20160921162135-0005/0 is now RUNNING
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - Registering block manager 172.29.30.215:51077 with 2004.6 MB RAM, BlockManagerId(driver, 172.29.30.215, 51077)
[Gogo shell] INFO org.apache.spark.storage.BlockManagerMaster - Registered BlockManager BlockManagerId(driver, 172.29.30.215, 51077)
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@18e8216b{/metrics/json,null,AVAILABLE}
[Gogo shell] INFO org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
[Gogo shell] WARN org.apache.spark.SparkContext - Use an existing SparkContext, some configuration may not take effect.
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5b855aa1{/SQL,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5df316cb{/SQL/json,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@78a44f1f{/SQL/execution,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7e3bbe8{/SQL/execution/json,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1476f413{/static/sql,null,AVAILABLE}
[Gogo shell] INFO org.apache.spark.sql.internal.SharedState - Warehouse path is 'file:/Users/matteorulli/git/nonono/com.nonono.osgi.example.int/com.nonono.osgi.example.int.etl/spark-warehouse'.
[Gogo shell] INFO org.apache.spark.sql.execution.SparkSqlParser - Parsing command: SHOW TABLES
java.lang.AssertionError: assertion failed: unsafe symbol DeveloperApi (child of package annotation) in runtime reflection universe
    at scala.reflect.internal.Symbols$Symbol.<init>(Symbols.scala:184)
    at scala.reflect.internal.Symbols$TypeSymbol.<init>(Symbols.scala:3009)
    at scala.reflect.internal.Symbols$ClassSymbol.<init>(Symbols.scala:3201)
    at scala.reflect.internal.Symbols$StubClassSymbol.<init>(Symbols.scala:3496)
    at scala.reflect.internal.Symbols$Symbol.newStubSymbol(Symbols.scala:498)
    at scala.reflect.internal.pickling.UnPickler$Scan.readExtSymbol$1(UnPickler.scala:258)
    at scala.reflect.internal.pickling.UnPickler$Scan.readSymbol(UnPickler.scala:284)
    at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolRef(UnPickler.scala:649)
    at scala.reflect.internal.pickling.UnPickler$Scan.readType(UnPickler.scala:417)
    at scala.reflect.internal.pickling.UnPickler$Scan$$anonfun$readTypeRef$1.apply(UnPickler.scala:658)
    at scala.reflect.internal.pickling.UnPickler$Scan$$anonfun$readTypeRef$1.apply(UnPickler.scala:658)
    at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:179)
    at scala.reflect.internal.pickling.UnPickler$Scan.readTypeRef(UnPickler.scala:658)
    at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotationInfo(UnPickler.scala:492)
    at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolAnnotation(UnPickler.scala:515)
    at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:97)
    at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:38)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:619)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply$mcV$sp(SymbolLoaders.scala:28)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:25)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.info(SynchronizedSymbols.scala:189)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:286)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
    at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
    at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
    at scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
    at scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
    at scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
    at scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
    at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
    at scala.reflect.internal.Types$Type.member(Types.scala:600)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
    at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:77)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:122)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
    at org.apache.spark.sql.types.StringType$$typecreator1$1.apply(StringType.scala:37)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
    at org.apache.spark.sql.types.AtomicType.<init>(AbstractDataType.scala:136)
    at org.apache.spark.sql.types.StringType.<init>(StringType.scala:32)
    at org.apache.spark.sql.types.StringType$.<init>(StringType.scala:48)
    at org.apache.spark.sql.types.StringType$.<clinit>(StringType.scala)
    at org.apache.spark.sql.execution.command.ShowTablesCommand.<init>(tables.scala:556)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:123)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:53)
    at org.apache.spark.sql.catalyst.parser.SqlBaseParser$ShowTablesContext.accept(SqlBaseParser.java:844)
    at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:63)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:54)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:82)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:46)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
    at com.nonono.osgi.example.int.etl.command.EtlCommand.etl(EtlCommand.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Console.run(Console.java:62)
    at org.apache.felix.gogo.shell.Shell.console(Shell.java:203)
    at org.apache.felix.gogo.shell.Shell.gosh(Shell.java:128)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Activator.run(Activator.java:75)
    at java.lang.Thread.run(Thread.java:745)
java.lang.ExceptionInInitializerError
    at org.apache.spark.sql.execution.command.ShowTablesCommand.<init>(tables.scala:556)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:123)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:53)
    at org.apache.spark.sql.catalyst.parser.SqlBaseParser$ShowTablesContext.accept(SqlBaseParser.java:844)
    at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:63)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:54)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:82)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:46)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
    at com.nonono.osgi.example.int.etl.command.EtlCommand.etl(EtlCommand.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Console.run(Console.java:62)
    at org.apache.felix.gogo.shell.Shell.console(Shell.java:203)
    at org.apache.felix.gogo.shell.Shell.gosh(Shell.java:128)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Activator.run(Activator.java:75)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: error reading Scala signature of org.apache.spark.sql.package: assertion failed: unsafe symbol DeveloperApi (child of package annotation) in runtime reflection universe
    at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:46)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:619)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply$mcV$sp(SymbolLoaders.scala:28)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:25)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.info(SynchronizedSymbols.scala:189)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:286)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
    at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
    at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
    at scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
    at scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
    at scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
    at scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
    at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
    at scala.reflect.internal.Types$Type.member(Types.scala:600)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
    at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:77)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:122)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
    at org.apache.spark.sql.types.StringType$$typecreator1$1.apply(StringType.scala:37)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
    at org.apache.spark.sql.types.AtomicType.<init>(AbstractDataType.scala:136)
    at org.apache.spark.sql.types.StringType.<init>(StringType.scala:32)
    at org.apache.spark.sql.types.StringType$.<init>(StringType.scala:48)
    at org.apache.spark.sql.types.StringType$.<clinit>(StringType.scala)
    ... 49 more
like image 598
matteo rulli Avatar asked Sep 21 '16 15:09

matteo rulli

1 Answers

It turned out that the spark jar containing the DeveloperApi class was missing from the classpath and the class did not get included in the spark bundle.

So, in this case, unsafe symbol DeveloperApi in runtime reflection universe traduces to the more prosaic ClassNotFoundException DeveloperApi: in term of exceptions messages java seams more concise than scala, after all.

like image 142
matteo rulli Avatar answered Oct 02 '22 00:10

matteo rulli
Sign in to Comment

Related questions

What is the equivalent of the retired mvn eclipse:eclipse in m2e?
How to connect to a running bigtable emulator from java
Can't create handler inside thread that has not called Looper.prepare() on alert dialog thread
why hibernate not set foreign key in the owner side of OneToMany
Spring boot application in windows server
Spring Boot 1.4 - TestRestTemplate Unsatisfied dependency exception
Java - Access-Control-Allow-Origin Multiple Origin Domains?
Java generics variable <T> value
ModelMapper converter - not working
Implicitly qualified autowiring in Spring
Sonar rule S2699: Not all asserts are recognized as valid assertions
Spring Data JPA. Pagination for child entities
Lombok Eclipse issue: "Requesting Java AST from selection"
Scala package object getClass
Lombok javafx properties
How to prevent integer input in Java?
How to Group By using Month from date stored as millisecond Postgres
Firebase retrieve Data from child by key
Semaphore - why my threads are running one after the other and not in concurrent?
Nifi "PutSQL" Out of bounds exception

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!