Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python requests module is very slow on specific machine

I've experienced too slow execution of Python requests on some machines and with specific user while other tools (for instance curl) are quite fast. Strange thing is that if run the script as another user then it runs as expected. If I run the script on my machine (both Windows or Linux) then it runs as expected too. Problematic machines are Windows 2008 servers on Hyper-V. I usually use POST request but both POST and GET are affected. For the demonstration I've created simple script with GET request. All requests take about 4.8s but it should take about 0.03s (virtual machines are not so powerful).

[imports and logging configuration omitted]

log.info("Started ...")

start = time.time()
response1 = requests.get("http://10.50.30.216:8080/sps/api/version")
assert response1.status_code == codes.OK
log.info("Using requests: %.3fs" % (time.time() - start))

start = time.time()
conn = httplib.HTTPConnection("10.50.30.216:8080")
conn.request("GET", "/sps/api/version")
response2 = conn.getresponse()
assert response2.status == codes.OK
log.info("Using httplib: %.3fs" % (time.time() - start))

log.info("Finished ...")

Output when logged as problematic user (unfortunately I must use that user). See that requests module waits 4.523s before opening a connection while httplib module proceeds immediately.

2015-09-11 14:50:00,832 - INFO - myscript - Started ...
2015-09-11 14:50:05,355 - INFO - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): 10.50.30.216
2015-09-11 14:50:05,364 - DEBUG - requests.packages.urllib3.connectionpool - "GET /sps/api/version HTTP/1.1" 200 None
2015-09-11 14:50:05,365 - INFO - myscript - Using requests: 4.533s
2015-09-11 14:50:05,374 - INFO - myscript - Using httplib: 0.008s
2015-09-11 14:50:05,375 - INFO - myscript - Finished ...

Output when logged as another user. Note that both users have Administrator privileges but the second user is only temporary and only on one machine so I can't use solve this issue by switching users.

2015-09-11 14:57:45,789 - INFO - myscript - Started ...
2015-09-11 14:57:45,799 - INFO - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): 10.50.30.216
2015-09-11 14:57:45,806 - DEBUG - requests.packages.urllib3.connectionpool - "GET /sps/api/version HTTP/1.1" 200 None
2015-09-11 14:57:45,809 - INFO - myscript - Using requests: 0.021s
2015-09-11 14:57:45,815 - INFO - myscript - Using httplib: 0.004s
2015-09-11 14:57:45,815 - INFO - myscript - Finished ...

I've read Python requests are slow #1 and Python requests are slower thann curl but it does not apply to my problem.

like image 710
Sobik Avatar asked Sep 10 '15 15:09

Sobik


1 Answers

For anyone reading this and using localhost in their URL, I resolved this issue by changing it to 127.0.0.1.

If this resolves the problem, it's a DNS problem, and it's not a requests problem.

like image 56
Casper Hansen Avatar answered Oct 02 '22 00:10

Casper Hansen