Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to check the Spark version

as titled, how do I know which version of spark has been installed in the CentOS?

The current system has installed cdh5.1.0.

like image 507
HappyCoding Avatar asked Apr 17 '15 03:04

HappyCoding


People also ask

How do I know if Apache Spark is installed?

To test if your installation was successful, open Command Prompt, change to SPARK_HOME directory and type bin\pyspark. This should start the PySpark shell which can be used to interactively work with Spark.

How do I check my Jupyter Spark version?

You can check your Spark setup by going to the /bin directory inside {YOUR_SPARK_DIRECTORY} and running the spark-shell –version command. Here you can see which version of Spark you have and which versions of Java and Scala it is using. That's it!

What is the latest Apache Spark version?

Spark 1.4. 1! This is a maintenance release that includes contributions from 85 developers. Spark 1.4. 1 includes fixes across several areas of Spark, including the DataFrame API, Spark Streaming, PySpark, Spark SQL, and MLlib.


2 Answers

If you use Spark-Shell, it appears in the banner at the start.

Programatically, SparkContext.version can be used.

like image 197
Shyamendra Solanki Avatar answered Sep 21 '22 11:09

Shyamendra Solanki


Open Spark shell Terminal, run sc.version

enter image description here

like image 21
Venu A Positive Avatar answered Sep 22 '22 11:09

Venu A Positive