Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

error: can't start new thread

I have a site that runs with follow configuration:

Django + mod-wsgi + apache

In one of user's request, I send another HTTP request to another service, and solve this by httplib library of python.

But sometimes this service don't get answer too long, and timeout for httplib doesn't work. So I creating thread, in this thread I send request to service, and join it after 20 sec (20 sec - is a timeout of request). This is how it works:

class HttpGetTimeOut(threading.Thread):     def __init__(self,**kwargs):         self.config = kwargs         self.resp_data = None         self.exception = None         super(HttpGetTimeOut,self).__init__()     def run(self):          h = httplib.HTTPSConnection(self.config['server'])         h.connect()         sended_data = self.config['sended_data']         h.putrequest("POST", self.config['path'])         h.putheader("Content-Length", str(len(sended_data)))         h.putheader("Content-Type", 'text/xml; charset="utf-8"')         if 'base_auth' in self.config:             base64string = base64.encodestring('%s:%s' % self.config['base_auth'])[:-1]             h.putheader("Authorization", "Basic %s" % base64string)         h.endheaders()          try:             h.send(sended_data)             self.resp_data = h.getresponse()         except httplib.HTTPException,e:             self.exception = e         except Exception,e:             self.exception = e 

something like this...

And use it by this function:

getting = HttpGetTimeOut(**req_config) getting.start() getting.join(COOPERATION_TIMEOUT) if getting.isAlive(): #maybe need some block     getting._Thread__stop()     raise ValueError('Timeout') else:     if getting.resp_data:         r = getting.resp_data     else:         if getting.exception:             raise ValueError('REquest Exception')         else:             raise ValueError('Undefined exception') 

And all works fine, but sometime I start catching this exception:

error: can't start new thread 

at the line of starting new thread:

getting.start() 

and the next and the final line of traceback is

File "/usr/lib/python2.5/threading.py", line 440, in start     _start_new_thread(self.__bootstrap, ()) 

And the answer is: What's happen?

Thank's for all, and sorry for my pure English. :)

like image 254
Oduvan Avatar asked Dec 02 '09 18:12

Oduvan


People also ask

Can t start new thread error?

The “can't start new thread” error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

How do you create a thread pool in Python?

futures module and its concrete subclass Executor, we can easily create a pool of threads. For this, we need to construct a ThreadPoolExecutor with the number of threads we want in the pool. By default, the number is 5. Then we can submit a task to the thread pool.

Why can't I start a new thread?

:) The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

Can't start new thread in Python?

Thank's for all, and sorry for my pure English. :) Show activity on this post. The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

Why is my threadpoolexecutor not working?

If you are using a ThreadPoolExecutor, the problem may be that your max_workers is higher than the threads allowed by your OS. It seems that the executor keeps the information of the last executed threads in the process table, even if the threads are already done.

Why does my application start multiple threads at once?

You are starting more threads than can be handled by your system. There is a limit to the number of threads that can be active for one process. Your application is starting threads faster than the threads are running to completion. If you need to start many threads you need to do it in a more controlled manner I would suggest using a thread pool.


2 Answers

The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

You should probably look at the number of threads you're creating; the maximum number you will be able to create will be determined by your environment, but it should be in the order of hundreds at least.

It would probably be a good idea to re-think your architecture here; seeing as this is running asynchronously anyhow, perhaps you could use a pool of threads to fetch resources from another site instead of always starting up a thread for every request.

Another improvement to consider is your use of Thread.join and Thread.stop; this would probably be better accomplished by providing a timeout value to the constructor of HTTPSConnection.

like image 72
Gabriel Reid Avatar answered Oct 01 '22 10:10

Gabriel Reid


You are starting more threads than can be handled by your system. There is a limit to the number of threads that can be active for one process.

Your application is starting threads faster than the threads are running to completion. If you need to start many threads you need to do it in a more controlled manner I would suggest using a thread pool.

like image 24
Tendayi Mawushe Avatar answered Oct 01 '22 10:10

Tendayi Mawushe