Python urllib2 Errors

Question

I have a long list of URLs I need to open for my service. Whenever I attempt to open this entire list, I receive many errors such as this when I initiate the program: [Errno 24] Too many open files. I am using urllib2 and gevent.

Does anyone have any solutions?

Thanks.

Answer 1

Use Gevent.pool to only open a certain amount of urls at a time then reuse those greenlets when thay have opened a url.