PYCHARM Error-- java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified

Question

I am getting the below error while running a pyspark program on PYCHARM, Error:

java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified ......

The interpreter is recognizing the python.exe file and I have added the Content root in project structure.

I got a similar issue while running the same program before in on windows command prompt and solved it using What is the right way to edit spark-env.sh before running spark-shell?

Answer 1

Before creating your spark session, set the following environment variables in your code:

import os
import sys
from pyspark.sql import SparkSession

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
spark = SparkSession.builder.getOrCreate()