Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why spark-shell fails with NullPointerException?

I try to execute spark-shell on Windows 10, but I keep getting this error every time I run it.

I used both latest and spark-1.5.0-bin-hadoop2.4 versions.

15/09/22 18:46:24 WARN Connection: BoneCP specified but not present in      CLASSPATH (or one of dependencies) 15/09/22 18:46:24 WARN Connection: BoneCP specified but not present in                 CLASSPATH (or one of dependencies) 15/09/22 18:46:27 WARN ObjectStore: Version information not found in     metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 15/09/22 18:46:27 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 15/09/22 18:46:27 WARN : Your hostname, DESKTOP-8JS2RD5 resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:103%net1, but we couldn't find any external IP address! java.lang.RuntimeException: java.lang.NullPointerException     at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)     at org.apache.spark.sql.hive.client.ClientWrapper.<init>    (ClientWrapper.scala:171)     at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala    :163)     at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)     at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:168)     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)     at java.lang.reflect.Constructor.newInstance(Unknown Source)     at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)     at $iwC$$iwC.<init>(<console>:9)     at $iwC.<init>(<console>:18)     at <init>(<console>:20)     at .<init>(<console>:24)     at .<clinit>(<console>)     at .<init>(<console>:7)     at .<clinit>(<console>)     at $print(<console>)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)     at java.lang.reflect.Method.invoke(Unknown Source)     at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)     at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)     at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)     at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)     at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)     at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)     at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)     at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)     at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132)     at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)     at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)     at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)     at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)     at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)     at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)     at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.sca      la:108)     at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$proc      ess$1.apply$mcZ$sp(SparkILoop.scala:991)     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$proc      ess$1.apply(SparkILoop.scala:945)     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$proc      ess$1.apply(SparkILoop.scala:945)     at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scal      a:135)     at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)     at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)     at org.apache.spark.repl.Main$.main(Main.scala:31)     at org.apache.spark.repl.Main.main(Main.scala)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)     at java.lang.reflect.Method.invoke(Unknown Source)     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)   Caused by: java.lang.NullPointerException     at java.lang.ProcessBuilder.start(Unknown Source)     at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)     at org.apache.hadoop.util.Shell.run(Shell.java:418)     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)     at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)     at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)     at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)     at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:559)     at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:534)    org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)     at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554) 

org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) ... 56 more

  <console>:10: error: not found: value sqlContext                import sqlContext.implicits._                 ^   <console>:10: error: not found: value sqlContext                import sqlContext.sql                 ^ 
like image 760
Nick Avatar asked Sep 22 '15 16:09

Nick


People also ask

How do I fix NullPointerException?

NullPointerException is thrown when a reference variable is accessed (or de-referenced) and is not pointing to any object. This error can be resolved by using a try-catch block or an if-else condition to check if a reference variable is null before dereferencing it.

What could be the reason if you are getting NullPointerException?

What Causes NullPointerException. The NullPointerException occurs due to a situation in application code where an uninitialized object is attempted to be accessed or modified. Essentially, this means the object reference does not point anywhere and has a null value.

Is NullPointerException a compile error?

NullPointerException is a RuntimeException . Runtime exceptions are critical and cannot be caught at compile time. They crash the program at run time if they are not handled properly.

Can we extend NullPointerException?

The Null Pointer Exception extends from the class RuntimeException. The hierarchy of NullPointerException is given below. As shown in the above hierarchy, Null Pointer Exception extends from the RuntimeException that inherits the Exception Class.


2 Answers

I used Spark 1.5.2 with Hadoop 2.6 and had similar problems. Solved by doing the following steps:

  1. Download winutils.exe from the repository to some local folder, e.g. C:\hadoop\bin.

  2. Set HADOOP_HOME to C:\hadoop.

  3. Create c:\tmp\hive directory (using Windows Explorer or any other tool).

  4. Open command prompt with admin rights.

  5. Run C:\hadoop\bin\winutils.exe chmod 777 /tmp/hive

With that, I am still getting some warnings, but no ERRORs and can run Spark applications just fine.

like image 86
max Avatar answered Sep 23 '22 13:09

max


I was facing a similar issue, got it resolved by putting the winutil inside bin folder. The Hadoop_home should be set as C:\Winutils and winutil to be placed in C:\Winutils\bin.

Windows 10 64 bit Winutils are available in https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin

Also ensure that command line has administrative access.

Refer https://wiki.apache.org/hadoop/WindowsProblems

like image 22
user6567143 Avatar answered Sep 21 '22 13:09

user6567143