Need to ping 1000 urls every 2 minutes

Question

I have 1000 feed urls sitting in a MySQL database table. I need to do a http request to all these urls every 2 minutes. I wrote a php script to do that, but the script takes 5min 30sec to run.

I want to be able to finish all the 1000 requests in under a minute. Is there a way to run multiple async processes to get the job done faster? Any help is appreciated. Thanks in advance.

Answer 1

Since your question is about sending http requests, not really ping, you can use Grequests (Requests+gevent) to do it easily and fast (in my experience seconds for a couple hundred url requests):

import grequests

urls = [
    'http://www.python.org',
    'http://python-requests.org',
    'http://www.google.com',
]
rs = (grequests.get(u) for u in urls)
grequests.map(rs)    # [<Response [200]>, <Response [200]>, <Response [200]>]

---

Your Php script takes 5 mins to run because it is synchronous code, which means that for every request you sent, you have to wait for response to arrive before moving onto sending the next request.

The trick here is not to wait (or block as many would call) for responses but go straight to make the next request, and you can achieve it easily with gevent(coroutine-based) or nodejs. You can read more on it here.