Menu
In the HTTP protocol you can send many requests in one socket using keep-alive and then receive the response from server at once, so that will significantly speed up whole process. Is there any way to do this in python requests lib? Or are there any other ways to speed this up that well using requests lib?
992
The HTTP keep-alive header maintains a connection between a client and your server, reducing the time needed to serve files. A persistent connection also reduces the number of TCP and SSL/TLS connection requests, leading to a drop in round trip time (RTT).
The keep alive timeout on the Message Processor allows a single TCP connection to send and receive multiple HTTP requests/responses from/to the backend server, instead of opening a new connection for every request/response pair.
Keep-Alive, also known as a persistent connection, is a communication pattern between a server and a client to reduce the HTTP request amount and speed up a web page. When Keep-Alive is turned on, the client and the server agree to keep the connection for subsequent requests or responses open.
Yes, there is. Use requests.Session and it will do keep-alive by default.
I guess I should include a quick example:
import logging import requests logging.basicConfig(level=logging.DEBUG) s = requests.Session() s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') s.get('http://httpbin.org/cookies/set/anothercookie/123456789') r = s.get("http://httpbin.org/cookies") print(r.text) You will note that these log message occur
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/sessioncookie/123456789 HTTP/1.1" 302 223 DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 55 DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/anothercookie/123456789 HTTP/1.1" 302 223 DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90 DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90 If you wait a little while, and repeat the last get call
INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: httpbin.org DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90 Note that it resets the dropped connection, i.e. reestablishing the connection to the server to make the new request.
174
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
