Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to know what is the reason for ClosedChannelExceptions with spark-shell in YARN client mode?

I have been trying to run spark-shell in YARN client mode, but I am getting a lot of ClosedChannelException errors. I am using spark 2.0.0 build for Hadoop 2.6.

Here are the exceptions :

$ spark-2.0.0-bin-hadoop2.6/bin/spark-shell --master yarn --deploy-mode client Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). 16/09/13 14:12:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/13 14:12:38 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 16/09/13 14:12:55 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED! 16/09/13 14:12:55 ERROR client.TransportClient: Failed to send RPC 7920194824462016141 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException java.nio.channels.ClosedChannelException 16/09/13 14:12:55 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalStateException: Spark context stopped while waiting for backend     at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)     at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)     at org.apache.spark.SparkContext.<init>(SparkContext.scala:549)     at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)     at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)     at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)     at scala.Option.getOrElse(Option.scala:121)     at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)     at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)     at $line3.$read$$iw$$iw.<init>(<console>:15)     at $line3.$read$$iw.<init>(<console>:31)     at $line3.$read.<init>(<console>:33)     at $line3.$read$.<init>(<console>:37)     at $line3.$read$.<clinit>(<console>)     at $line3.$eval$.$print$lzycompute(<console>:7)     at $line3.$eval$.$print(<console>:6)     at $line3.$eval.$print(<console>)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:498)     at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)     at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)     at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)     at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)     at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)     at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)     at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)     at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)     at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)     at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)     at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)     at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)     at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)     at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)     at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)     at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:94)     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)     at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)     at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)     at org.apache.spark.repl.Main$.doMain(Main.scala:68)     at org.apache.spark.repl.Main$.main(Main.scala:51)     at org.apache.spark.repl.Main.main(Main.scala)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:498)     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/09/13 14:12:55 WARN netty.NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 1 attempts org.apache.spark.SparkException: Exception thrown in awaitResult     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)     at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)     at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)     at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)     at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)     at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)     at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply$mcV$sp(YarnSchedulerBackend.scala:271)     at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)     at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)     at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)     at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)     at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Failed to send RPC 7920194824462016141 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException     at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)     at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)     at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)     at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)     at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)     at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)     at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)     at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)     at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)     at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)     at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)     at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)     at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)     ... 1 more Caused by: java.nio.channels.ClosedChannelException java.lang.IllegalStateException: Spark context stopped while waiting for backend   at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)   at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)   at org.apache.spark.SparkContext.<init>(SparkContext.scala:549)   at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)   at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)   at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)   at scala.Option.getOrElse(Option.scala:121)   at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)   at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)   ... 47 elided <console>:14: error: not found: value spark        import spark.implicits._               ^ <console>:14: error: not found: value spark        import spark.sql               ^ Welcome to       ____              __      / __/__  ___ _____/ /__     _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 2.0.0       /_/  Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101) Type in expressions to have them evaluated. Type :help for more information.  scala> 16/09/13 14:12:59 ERROR client.TransportClient: Failed to send RPC 5797372389565173518 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException 16/09/13 14:12:59 WARN netty.NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 2 attempts org.apache.spark.SparkException: Exception thrown in awaitResult     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)     at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)     at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)     at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)     at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)     at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)     at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply$mcV$sp(YarnSchedulerBackend.scala:271)     at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)     at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)     at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)     at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)     at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Failed to send RPC 5797372389565173518 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException     at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)     at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)     at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)     at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)     at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)     at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)     at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)     at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)     at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)     at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)     at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)     at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)     at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)     ... 1 more Caused by: java.nio.channels.ClosedChannelException 
like image 431
aks Avatar asked Sep 13 '16 10:09

aks


People also ask

How do you start the Spark Shell in yarn mode?

Launching Spark on YARNEnsure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster. These configs are used to write to HDFS and connect to the YARN ResourceManager.

What is Spark in yarn mode?

Spark supports two modes for running on YARN, “yarn-cluster” mode and “yarn-client” mode. Broadly, yarn-cluster mode makes sense for production jobs, while yarn-client mode makes sense for interactive and debugging uses where you want to see your application's output immediately.


1 Answers

Reason is association with yarn cluster may be lost due to the Java 8 excessive memory allocation issue: https://issues.apache.org/jira/browse/YARN-4714

You can force YARN to ignore this by setting up the following properties in yarn-site.xml

<property>     <name>yarn.nodemanager.pmem-check-enabled</name>     <value>false</value> </property>  <property>     <name>yarn.nodemanager.vmem-check-enabled</name>     <value>false</value> </property> 

Thanks to simplejack, Reference from Spark Pi Example in Cluster mode with Yarn: Association lost

like image 110
aks Avatar answered Sep 27 '22 21:09

aks