Output reason for Python crash

Question

I have an application which polls a bunch of servers every few minutes. To do this, it spawns one thread per server to poll (15 servers) and writes back the data to an object:

import requests

class ServerResults(object):
    def __init__(self):
        self.results = []

    def add_server(some_argument):
        self.results.append(some_argument)

servers = ['1.1.1.1', '1.1.1.2']
results = ServerResults()

for s in servers:
    t = CallThreads(poll_server, s, results)
    t.daemon = True
    t.start()

def poll_server(server, results):
    response = requests.get(server, timeout=10)
    results.add_server(response.status_code);

The CallThreads class is a helper function to call a function (in this case poll_server() with arguments (in this case s and results), you can see the source at my Github repo of Python utility functions. Most of the time this works fine, however sometimes a thread intermittently hangs. I'm not sure why, since I am using a timeout on the GET request. In any case, if the thread hangs then the hung threads build up over the course of hours or days, and then Python crashes:

  File "/usr/lib/python2.7/threading.py", line 495, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

Exception in thread Thread-575 (most likely raised during interpreter shutdown)
Exception in thread Thread-1671 (most likely raised during interpreter shutdown)
Exception in thread Thread-831 (most likely raised during interpreter shutdown)

How might I deal with this? There seems to be no way to kill a blocking thread in Python. This application needs to run on a Raspberry Pi, so large libraries such as twisted won't fit, in fact I need to get rid of the requests library as well!

Answer 1

As far as I can tell, a possible scenario is that when a thread "hangs" for one given server, it will stay there "forever". Next time you query your servers another thread is spawned (_start_new_thread), up to the point where Python crashes.

Probably not your (main) problem, but you should:

• use a thread pool - this won't stress the limited resources of your your system as much as spawning new threads again and again.
• check that you use a "thread-compatible" mechanism to handle concurrent access to results. Maybe a semaphore or mutex to lock atomic portions of your code. Probably better would be a dedicated data structure such as a queue.

---

Concerning the "hang" per se -- beware that the timeout argument while "opening a URL" (urlopen) is related to the time-out for establishing the connection. Not for downloading the actual data:

The optional timeout parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS and FTP connections.