Start a TCPServer with ThreadingMixIn again in code directly after shutting it down. (Gives `Address already in use`)

Question

I try to program a TCPServer with threads (ThreadingMixIn) in Python. The problem is that I can't shut it down properly as I get the socket.error: [Errno 48] Address already in use when I try to run it again. This is a minimal example of the python code that triggers the problem:

import socket
import threading
import SocketServer

class FakeNetio230aHandler(SocketServer.BaseRequestHandler):

    def send(self,message):
        self.request.send(message+N_LINE_ENDING)

    def handle(self):
        self.request.send("Hello\n")

class FakeNetio230a(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
    def __init__(self, server_address, RequestHandlerClass):
        self.allow_reuse_address = True
        SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass)

if __name__ == '__main__':
    for i in range(2):
        fake_server = FakeNetio230a(("", 1234), FakeNetio230aHandler)
        server_thread = threading.Thread(target=fake_server.serve_forever)
        server_thread.setDaemon(True)
        server_thread.start()
        # might add some client connection here
        fake_server.shutdown()

All the main code should do is to start the server, shut it down and run it again. But it triggers the error stated above because the socket has not been released after the first shutdown.

I thought that setting self.allow_reuse_address = True could solve the problem, but that did not work. When the python program finishes I can run it again straight away and it can start the server once (but again not twice).
However the problem is gone when I randomize the port (replace 1234 by 1234+i for example) as no other server is listening on that address.

There is a similar SO Q Shutting down gracefully from ThreadingTCPServer but the solution (set allow_reuse_address to True does not work for my code and I don't use ThreadingTCPServer).

How do I have to modify my code in order to be able to start the server twice in my code?

Some more information: The reason why I'm doing this is that I want to run some unit tests for my python project. This requires to provide a (fake) server that my software should to connect to.

edit:
I just found the most correct answer to my problem: I have to add fake_server.server_close() at the end of my main execution code (right after fake_server.shutdown()). I found it in the source file of the TCPServer implementation. All it does is self.socket.close().

Answer 1

Somehow, fake_server doesn't unbind when you assign to it (in first line in for statement).

To fix that, just remove fake_server at the end of loop:

        del fake_server # force server to unbind

Answer 2

This post helped me get over the un-closed socket problem. I had the same problem and wanted to post here my simple implementation for TCP server class (and client method).

I made a TCPThreadedServer class. In order to use it is needed to be inherited, and the method process(msg) must be overridden. the overridden method invokes every time the server gets a message msg, and if it returns a not None object, it will be returned as string to the connected client.

from SocketServer import TCPServer, StreamRequestHandler, ThreadingMixIn
import threading

class TCPThreadedServer(TCPServer, ThreadingMixIn):
    class RequstHandler(StreamRequestHandler):
       def handle(self):
           msg = self.rfile.readline().strip()
           reply = self.server.process(msg)
           if reply is not None:
               self.wfile.write(str(reply) + '\n')

    def __init__(self, host, port, name=None):
        self.allow_reuse_address = True
        TCPServer.__init__(self, (host, port), self.RequstHandler)
        if name is None: name = "%s:%s" % (host, port)
        self.name = name
        self.poll_interval = 0.5

    def process(self, msg):
        """
        should be overridden
        process a message
        msg    - string containing a received message
        return - if returns a not None object, it will be sent back 
                 to the client.
        """
        raise NotImplemented

    def serve_forever(self, poll_interval=0.5):
        self.poll_interval = poll_interval
        self.trd = threading.Thread(target=TCPServer.serve_forever,
                                    args = [self, self.poll_interval],
                                    name = "PyServer-" + self.name)
        self.trd.start()

    def shutdown(self):
        TCPServer.shutdown(self)
        TCPServer.server_close(self)
        self.trd.join()
        del self.trd

I found it quite easy to use:

class EchoServerExample(TCPThreadedServer):
    def __init__(self):
        TCPThreadedServer.__init__(self, "localhost", 1234, "Server")

    def process(self, data):
        print "EchoServer Got: " + data
        return str.upper(data)

for i in range(10):
    echo = EchoServerExample()
    echo.serve_forever()

    response = client("localhost", 1234, "hi-%i" % i)
    print "Client received: " + response

    echo.shutdown()

I used the method: import socket

def client(ip, port, msg, recv_len=4096, 
           timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
    msg = str(msg)
    response = None
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    try:
        sock.connect((ip, port))
        if timeout != socket._GLOBAL_DEFAULT_TIMEOUT:
            sock.settimeout(timeout)
        sock.send(msg + "\n")
        if recv_len > 0:
            response = sock.recv(recv_len)
    finally:
        sock.close()
        return response

Enjoy it!

Answer 3

Change your FakeNetio230a definition to this:

class FakeNetio230a(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
    def __init__(self, server_address, RequestHandlerClass):
        self.allow_reuse_address = True
        SocketServer.TCPServer.__init__(self,
                                        server_address,
                                        RequestHandlerClass,
                                        False)  # do not implicitly bind

Then, add these two lines in your entry point below your FakeNetio230a instantiation:

fake_server.server_bind()    # explicitly bind
fake_server.server_activate()   # activate the server

Here's an example:

if __name__ == '__main__':
    for i in range(2):
        fake_server = FakeNetio230a(("", 1234), FakeNetio230aHandler)
        fake_server.server_bind()    # explicitly bind
        fake_server.server_activate()   # activate the server
        server_thread = threading.Thread(target=fake_server.serve_forever)
        server_thread.setDaemon(True)
        server_thread.start()
        # might add some client connection here
        fake_server.shutdown()