Multi Thread Requests Python3

Question

I have researched a lot on this topic but the problem is am not able to figure out how to send multi-threading post requests using python3

names = ["dfg","dddfg","qwed"]

for name in names :
    res = requests.post(url,data=name)
    res.text 

Here I want to send all these names and I want to use multi threading to make it faster.

Answer 1

Solution 1 - concurrent.futures.ThreadPoolExecutor fixed number of threads

Using a custom function (request_post) you can do almost anything.

import concurrent
import requests

def request_post(url, data):
    return requests.post(url, data=data)

with concurrent.futures.ThreadPoolExecutor() as executor: # optimally defined number of threads
    res = [executor.submit(request_post, url, data) for data in names]
    concurrent.futures.wait(res)

res will be list of request.Response for each request made wrapped on Future instances. To access the request.Response you need to use res[index].result() where index size is len(names).

Future objects give you better control on the responses received, like if it completed correctly or there was an exception or time-out etc. More about here

You don't take risk of problems related to high number of threads (solution 2).

---

Solution 2 - multiprocessing.dummy.Pool and spawn one thread for each request

Might be usefull if you are not requesting a lot of pages and also or if the response time is quite slow.

from multiprocessing.dummy import Pool as ThreadPool
import itertools
import requests

with ThreadPool(len(names)) as pool: # creates a Pool of 3 threads 
    res = pool.starmap(requests.post(itertools.repeat(url),names))

pool.starmap - is used to pass (map) multiple arguments to one function (requests.post) that is gonna be called by a list of Threads (ThreadPool). It will return a list of request.Response for each request made.

intertools.repeat(url) is needed to make the first argument be repeated the same number of threads being created.

names is the second argument of requests.post so it's gonna work without needing to explicitly use the optional parameter data. Its len must be the same of the number of threads being created.

This code will not work if you needed to call another parameter like an optional one