No module named 'resource' installing Apache Spark on Windows

Question

I am trying to install apache spark to run locally on my windows machine. I have followed all instructions here https://medium.com/@loldja/installing-apache-spark-pyspark-the-missing-quick-start-guide-for-windows-ad81702ba62d.

After this installation I am able to successfully start pyspark, and execute a command such as

textFile = sc.textFile("README.md")

When I then execute a command that operates on textFile such as

textFile.first()

Spark gives me the error 'worker failed to connect back', and I can see an exception in the console coming from worker.py saying 'ModuleNotFoundError: No module named resource'. Looking at the source file I can see that this python file does indeed try to import the resource module, however this module is not available on windows systems. I understand that you can install spark on windows so how do I get around this?

Answer 1

I struggled the whole morning with the same problem. Your best bet is to downgrade to Spark 2.3.2