Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python: httplib getresponse issues many recv() calls

getresponse issues many recv calls while reading header of an HTML request. It actually issues recv for each byte which results in many system calls. How can it be optimized?

I verified on an Ubuntu machine with strace dump.

sample code:

conn = httplib.HTTPConnection("www.python.org")
conn.request("HEAD", "/index.html")
r1 = conn.getresponse()

strace dump:

sendto(3, "HEAD /index.html HTTP/1.1\r\nHost:"..., 78, 0, NULL, 0) = 78
recvfrom(3, "H", 1, 0, NULL, NULL)      = 1
recvfrom(3, "T", 1, 0, NULL, NULL)      = 1
recvfrom(3, "T", 1, 0, NULL, NULL)      = 1
recvfrom(3, "P", 1, 0, NULL, NULL)      = 1
recvfrom(3, "/", 1, 0, NULL, NULL)      = 1
...
like image 441
nik_kgp Avatar asked Oct 21 '22 18:10

nik_kgp


1 Answers

r = conn.getresponse(buffering=True)

On Python 3.1+ there is no buffering parameter (it is default).

like image 64
jfs Avatar answered Nov 02 '22 23:11

jfs