How to check version of Spark and Hadoop in AWS glue?

Question

I am using AWS with (Basic support plan). I want to know which version of Spark and Hadoop (HDFS) is getting used in AWS glue jobs. So that I can setup the same environment in my local machine for development.

Or if i get to know version of Spark then corresponding which version of Hadoop is used by AWS glue jobs or vice-versa.

As I am using Basic Support plan. So, I can't raise case to support center. Any idea where I can check in AWS glue jobs... Spark and Hadoop version?

Any kind of help and suggestion is appreciated. Thanks!

Answer 1

If you are using Spark > 2.0 then

1.In Pyspark:

Get Spark version:

print("Spark Version:" + spark.version)

In spark < 2.0:

sc.version

Get Hadoop version:

print("Hadoop version: " + sc._gateway.jvm.org.apache.hadoop.util.VersionInfo.getVersion())

2.In Scala:

Spark Version:

println ("Spark Version:" + spark.version)

in spark < 2.0:

sc.version

Hadoop version:

println("Hadoop version: " + org.apache.hadoop.util.VersionInfo.getVersion())