Python: multiprocessing and requests

Question

Following is the snippet of code I am running to use multiprocessing which fires HTTP request in parallel. After the running on console it is getting hung at "requests.get(url)" and neither proceeding ahead nor throwing an error.

def echo_100(q):
    ...
    print "before"
    r = requests.get(url)
    print "after"
    ...
    q.put(r)

q = multiprocessing.Queue()
p = multiprocessing.Process(target=echo_100,args=(q))
p.start()
p.join()
resp = q.get()

Answer 1

On Mac OS, there seem to be some bugs reading proxy settings from the operating system. I don't know the exact details, but it sometimes causes requests to hang when using multiprocessing. You could try to circumvent the problem by disabling OS proxies entirely, like this:

session = requests.Session()
session.trust_env = False  # Don't read proxy settings from OS
r = session.get(url)

That fixed it for me.