Python version running on EMR 6.8

Question

What version of Python does EMR 6.8 support?

It looks like previous versions of EMR supported Python 3.7 which will be deprecated in 6 months.

EMR 6.8 runs Spark 3.3.0 which supports Python up to 3.9.

I can't find the answer on the official release guide: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-680-release.html

Answer 1

Since this question was asked, they have updated the page at https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-6.x.html with the information you are looking for. There is now a line item for included Python version.

emr-6.8.0 includes Python versions 2.7, 3.7.

As of today, the latest EMR version is emr-6.14.0 which still only includes Python 3.7.

Answer 2

• EMR 6.8.0 aligns with Spark/PySpark 3.3.0, according to https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-6x.html
• PySpark 3.3.0 supports Python 3.7 and up, according to https://spark.apache.org/docs/3.3.0/api/python/getting_started/install.html#python-version-supported
• and up certainly means 3.7, 3.8, 3.9 and 3.10, as those are the stable and currently supported releases of Python according to their respective release schedule PEPs:
  • 3.10 PEP 619
  • 3.9 PEP 596
  • 3.8 PEP 569
  • 3.7 PEP 537

Which Python is available to you is a question about which compute you're using in EMR to run PySpark. If you want another Python, you may need to pick another AMI, a different container image, or talk to AWS directly.