ConnectionRefusedError - Python pyspark

Question

I tried to run this simple Spark session creation command in my Jupyter notebook -

spark = SparkSession.builder.getOrCreate()

But I am continuously getting the following error:

---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)
Cell In[22], line 1
----> 1 spark = SparkSession.builder.getOrCreate()

File C:\Python\lib\site-packages\pyspark\sql\session.py:503, in SparkSession.Builder.getOrCreate(self)
    500     session = SparkSession(sc, options=self._options)
    501 else:
    502     getattr(
--> 503         getattr(session._jvm, "SparkSession$"), "MODULE$"
    504     ).applyModifiableSettings(session._jsparkSession, self._options)
    505 return session

File C:\Python\lib\site-packages\py4j\java_gateway.py:1712, in JVMView.__getattr__(self, name)
   1709 if name == UserHelpAutoCompletion.KEY:
   1710     return UserHelpAutoCompletion()
-> 1712 answer = self._gateway_client.send_command(
   1713     proto.REFLECTION_COMMAND_NAME +
   1714     proto.REFL_GET_UNKNOWN_SUB_COMMAND_NAME + name + "\n" + self._id +
   1715     "\n" + proto.END_COMMAND_PART)
   1716 if answer == proto.SUCCESS_PACKAGE:
   1717     return JavaPackage(name, self._gateway_client, jvm_id=self._id)

File C:\Python\lib\site-packages\py4j\java_gateway.py:1036, in GatewayClient.send_command(self, command, retry, binary)
   1015 def send_command(self, command, retry=True, binary=False):
   1016     """Sends a command to the JVM. This method is not intended to be
   1017        called directly by Py4J users. It is usually called by
   1018        :class:`JavaMember` instances.
   (...)
   1034      if `binary` is `True`.
   1035     """
-> 1036     connection = self._get_connection()
   1037     try:
   1038         response = connection.send_command(command)

File C:\Python\lib\site-packages\py4j\clientserver.py:284, in JavaClient._get_connection(self)
    281     pass
    283 if connection is None or connection.socket is None:
--> 284     connection = self._create_new_connection()
    285 return connection

File C:\Python\lib\site-packages\py4j\clientserver.py:291, in JavaClient._create_new_connection(self)
    287 def _create_new_connection(self):
    288     connection = ClientServerConnection(
    289         self.java_parameters, self.python_parameters,
    290         self.gateway_property, self)
--> 291     connection.connect_to_java_server()
    292     self.set_thread_connection(connection)
    293     return connection

File C:\Python\lib\site-packages\py4j\clientserver.py:438, in ClientServerConnection.connect_to_java_server(self)
    435 if self.ssl_context:
    436     self.socket = self.ssl_context.wrap_socket(
    437         self.socket, server_hostname=self.java_address)
--> 438 self.socket.connect((self.java_address, self.java_port))
    439 self.stream = self.socket.makefile("rb")
    440 self.is_connected = True

ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

However, this command was working perfectly when i ran it for the first time, but after few hours, the above error has become persistent.

Answer 1

I had the same problem. A restart of the running kernel fixed the problem for me.

It seems that this happens when I put my PC to sleep mode for a long time and then try to continue using the Jupyter notebook.