Which Scala version does Spark 2.4.3 uses?

Question

I installed Scala(Version 2.12.8) and Spark(2.4.3) on my Mac OS from homebrew. I already have Java 1.8 installed on my machine.

When I launch spark-shell, I see the logo says:

Spark version 2.4.3, Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)

Why does is says Scala version 2.11.12 instead of Scala(Version 2.12.8) which installed on my machine?

Does Spark 2.4.3 come with Scala 2.11.12?

Thanks.

Answer 1

As stated in the release notes:

Spark 2.4.3 is a maintenance release containing stability fixes. This release is based on the branch-2.4 maintenance branch of Spark. We strongly recommend all 2.4 users to upgrade to this stable release.

Note that 2.4.3 switched the default Scala version from Scala 2.12 to Scala 2.11, which is the default for all the previous 2.x releases except 2.4.2. That means, the pre-built convenience binaries are compiled for Scala 2.11. Spark is still cross-published for 2.11 and 2.12 in Maven Central, and can be built for 2.12 from source.

Additionally Scala version you happen to have on your machine is completely irrelevant - Spark uses Scala version that has been used to compile it.