How to ping a range of IP addresses using multithreading

Question

Lets say that I have the following ip range "10.0.0.x". I need to loop on this range of ips - "10.0.0.1-255", ping each one, and check the response.

This is my code:

for ip in range(1, 256):
  fullIP = '10.0.0' + ip
  if(Ping(fullIP) == True):
    print(fullIP + ' responded')
  else:
    print(fullIP + ' did not respond')

This code works but unfortunately it is very slow.
I would like to make it more efficient through multi threading so i did the following:

def PingRange(start, end):
  for ip in range(start, end):
  fullIP = '10.0.0' + ip
  if(Ping(fullIP) == True):
    print(fullIP + ' responded')
  else:
    print(fullIP + ' did not respond')

try:
  thread.start_new_thread( PingRange, (1, 123))
  thread.start_new_thread( PingRange, (123, 256))
except:
  print "Error: unable to start thread"

This code also works, but it could work better and more universal.
If this code was written properly then I wouldn't just constantly create two threads; I would create as many threads as the operating system would allow me.
Some computers allow 8 threads, others allow only 4, and some don't even allow threading.

How can I make this program use the maximum amount of threads in Python?

Answer 1

This problem is well-suited for using a thread pool. A thread pool runs with a constant number of threads, takes work items (functions or methods), and executes those work items in its pool of threads. It has built-in queuing, so if you give one hundred work items to a pool of five threads, it will execute all one hundred items, but without ever running more than five concurrently.

There are two built-in thread pool options in Python (depending on what version you're using) - multiprocessing.dummy.Pool, and concurrent.futures.ThreadPoolExecutor. ThreadPoolExecutor is only built-in in Python 3.x, though a backport is available from PyPI. multiprocessing.dummy.Pool is available in 2.6+. Using a multiprocessing.dummy.Pool, your code becomes as simple as this:

import multiprocessing.dummy

def ping_range(start, end):
    num_threads = # Number of threads to run in the pool.
    p = multiprocessing.dummy.Pool(num_threads)
    p.map(ping, [10.0.0.x for x in range(start,end)])

if __name__ == "__main__":
    PingRange(0, 255)

The next question is what to use for num_threads. I think you're slightly misinformed about systems having a max number of allowable threads. You can create as many Thread objects as you want on any system, and nothing will actually stop you, but at a certain point you'll create so many threads that the system won't be able to handle it and performance will start to get worse, rather than better.

The rule of thumb for CPU-bound applications (meaning that it primarily requires the CPU to do work) is to run as many threads as there are CPUs. However, this ping operation is I/O-bound, meaning most of the work is sending the ping requests to external systems and then waiting for a response, which doesn't require the CPU to do anything. In those cases, it's usually ok to use more than the number of CPUs. We can be conservative and use 2 * number_of_cpus, though you could experiment with a larger number.

import multiprocessing
num_threads = 2 * multiprocessing.cpu_count()

Putting it all together:

import multiprocessing.dummy
import multiprocessing

def ping(ip):
   success = # Run the ping command here
   if success:
       print("{} responded".format(ip))
   else:
       print("{} did not respond".format(ip))
   return success

def ping_range(start, end):
    num_threads = 2 * multiprocessing.cpu_count()
    p = multiprocessing.dummy.Pool(num_threads)
    p.map(ping, [10.0.0.x for x in range(start,end)])

if __name__ == "__main__":
    ping_range(0, 255)