python-requests with multithreading

Question

I am working on creating a HTTP client which can generate hundreds of connections each second and send up to 10 requests on each of those connections. I am using threading so concurrency can be achieved. Here is my code:

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://20.20.1.2/tempurl.html')
            if response1.status_code == 200:
                client_notify('r')
        except(exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout) as Err:
            client_notify('F')
            break
        requestCounter += 1

def main():
    for q in range(connectionPerSec):
        s1 = requests.session()
        t1 = threading.Thread(target=generate_req, args=(s1,))
        t1.start()

Issues:

1. It is not scaling above 200 connections/sec with requestRate = 1. I ran other available HTTP clients on the same client machine and against the server, test runs fine and it is able to scale.

2. When requestRate = 10, connections/sec drops to 30. Reason: Not able to create targeted number of threads every second.

For issue #2, client machine is not able to create enough request sessions and start new threads. As soon as requestRate is set to more than 1, things start to fall apart. I am suspecting it has something to do with HTTP connection pooling which requests uses.

Please suggest what am I doing wrong here.

Answer 1

I wasn't able to get things to fall apart, however the following code has some new features:

1) extended logging, including specific per-thread information

2) all threads join()ed at the end to make sure the parent process doesntt leave them hanging

3) multithreaded print tends to interleave the messages, which can be unwieldy. This version uses yield so a future version can accept the messages and print them clearly.

source

import exceptions, requests, threading, time

requestRate = 1
connectionPerSec = 2


def client_notify(msg):
    return time.time(), threading.current_thread().name, msg

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://127.0.0.1/')
            if response1.status_code == 200:
                print client_notify('r')
        except (exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout):
            print client_notify('F')
            break
        requestCounter += 1

def main():
    for cnum in range(connectionPerSec):
        s1 = requests.session()
        th = threading.Thread(
            target=generate_req, args=(s1,),
            name='thread-{:03d}'.format(cnum),
        )
        th.start()

    for th in threading.enumerate():
        if th != threading.current_thread():
            th.join()


if __name__=='__main__':
    main()

output

(1407275951.954147, 'thread-000', 'r')
(1407275951.95479, 'thread-001', 'r')