AWS EMR 5.20 and Java version support

Question

I would like to know if Amazon EMR 5.20 AMI support Java 9?

I could not find the information here. I would be interested in using Spark 2.4.0 with JDK 9 and Scala 2.12.

https://docs.aws.amazon.com/fr_fr/emr/latest/ReleaseGuide/emr-release-5x.html https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html

Thank you.

Answer 1

First of all, you can not use Scala 2.12.x with Spark 2.4.0 (btw, there is experimental support, see the comments). As stated in the documentation, to use Spark 2.4.0, you will need to use a compatible Scala version which is 2.11.x family.

Secondly, from documentation, JDK9+ is compatible with Scala 2.11.12 but it is incomplete.

Finally, AWS EMR 5.20 uses Apache Spark 2.4.0. You can check JDK version from command line in your master node.

>java -version
openjdk version "1.8.0_191"
# "b" means "build"
# Version number is 8u191, which also means JDK 8 Update 191
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Or, you can directly run spark-shell command which will print both Scala (2.11.12) and JDK version (JDK8).

I hope it helps!

NOTE: Another doc also states that Java 8 is the default JVM for cluster instances for AWS EMR 5.0.0 or later.

Answer 2

amazon-emr has announced the availability of a new beta version which includes the latest apache-spark version at this time (2.4.4) and uses scala 2.12. Please check the following:

https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-emr-6-beta-2-adds-hive-3-with-llap-support-scala-212-with-spark-244/