Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Spark workers stopped after driver commanded a shutdown

Tags:

apache-spark

apache-spark-standalone

Basically, Master node also perform as a one of the slave. Once slave on master completed it called the SparkContext to stop and hence this command propagate to all the slaves which stop the execution in mid of the processing.

Error log in one of the worker:

INFO SparkHadoopMapRedUtil: attempt_201612061001_0008_m_000005_18112: Committed

INFO Executor: Finished task 5.0 in stage 8.0 (TID 18112). 2536 bytes result sent to driver

INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown

ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERMtdown

like image 690
pooshan Singh Avatar asked Dec 06 '16 10:12

pooshan Singh

People also ask

Why did my spark driver fail?

Since there is no host fail-over, each job submission results in the same failure. Since the Spark Master is responsible for starting the Driver, you can get more information from the Spark Master Logs indicating why the driver did not execute successfully. You can access the logs through the GUI or Command Line.

How to stop the Spark Streaming job gracefully?

There are currently two ways to stop the spark streaming job gracefully. The first way is to set spark.streaming.stopGracefullyOnShutdown parameter to be true (default is false). This parameter is introduced in Spark to solve the graceful shutdown issue. Developers do not need to call ssc.stop () in the codes any more.

How to stop the execution of a spark service in mid-processing?

Once slave on master completed it called the SparkContext to stop and hence this command propagate to all the slaves which stop the execution in mid of the processing. Error log in one of the worker:

What happens when Spark driver receives SIGTERM signal?

After the Spark driver received the SIGTERM signal, you should see the below messages in the log There is a catch though. Be default, the spark.yarn.maxAppAttempts parameter uses the default value from yarn.resourcemanager.am.max-attempts in YARN.

1 Answers

Check your resource manager user interface, in case you see any executor failed - it details about memory error. However if executor has not failed but still driver called for shut down - usually this is due to driver memory, please try to increase driver memory. Let me know how it goes.

like image 98
user500377 Avatar answered Sep 18 '22 11:09

user500377
Sign in to Comment

Related questions

Connect to spark through a SOCKS proxy
How do I submit a Spark jar to a EMR cluster?
Where to download documentation for Spark?
SparkR Error in sparkR.init(master="local") in RStudio
Multiple IP addresses and Host Names used by Spark Driver and Master
java.util.concurrent.RejectedExecutionException in Spark although driver/client has precisely same version as Server
Writing an RDD to multiple files in PySpark
Can sample weight be used in Spark MLlib Random Forest training?
Manually stopping Spark Workers
Spark Streaming: Broadcast variables, java.lang.ClassCastException
How to run custom Python script on Jupyter Notebook launch (to boot Spark)?
saveToCassandra with spark-cassandra connector throws java.lang.ClassCastException
How to load a PMML model?
How to distribute xgboost module for use in spark?
how to get two-hop neighbors in spark-graphx?
How a Spark executor runs multiple tasks?
Pyspark - Sum over multiple sparse vectors (CountVectorizer Output)
Can we use SizeEstimator.estimate for estimating size of RDD/DataFrame?
Slow Parquet write to HDFS using Spark
Spark performance enhancements by storing sorted Parquet files

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!