Can a requests library session object be used across greenlets safely in a gevented program?
EDIT - ADDING MORE EXPLANATION:
When a greenlet yields because it has made a socket call to send the request to the server, can the same socket (inside the shared session object) be used safely by another greenlet to send its own request?
END EDIT
I attempted to test this with the code posted here - https://gist.github.com/donatello/0b399d0353cb29dc91b0 - however I got no errors or unexpected results. However, this does not validate thread safety.
In the test, I use a shared session object to make lots of requests and try to see if the object gets the requests mixed up - it is kind of naive, but I do not get any exceptions.
For convenience, I am re-pasting the code here:
client.py
import gevent
from gevent.monkey import patch_all
patch_all()
import requests
import json
s = requests.Session()
def make_request(s, d):
r = s.post("http://127.0.0.1:5000", data=json.dumps({'value': d}))
if r.content.strip() != str(d*2):
print("Sent %s got %s" % (r.content, str(d*2)))
if r.status_code != 200:
print(r.status_code)
print(r.content)
gevent.joinall([
gevent.spawn(make_request, s, v)
for v in range(300)
])
server.py
from gevent.wsgi import WSGIServer
from gevent.monkey import patch_all
patch_all()
from flask import Flask
from flask import request
import time
import json
app = Flask(__name__)
@app.route('/', methods=['POST', 'GET'])
def hello_world():
d = json.loads(request.data)
return str(d['value']*2)
if __name__ == '__main__':
http_server = WSGIServer(('', 5000), app)
http_server.serve_forever()
Exact library versions:
requirements.txt
Flask==0.10.1
Jinja2==2.7.2
MarkupSafe==0.23
Werkzeug==0.9.4
argparse==1.2.1
gevent==1.0.1
greenlet==0.4.2
gunicorn==18.0
itsdangerous==0.24
requests==2.3.0
wsgiref==0.1.2
Is there some other test that can check greenlet thread safety? The requests documentation is not too clear on this point.
Unfortunately requests. Session() is not thread-safe. There are several strategies for making data accesses thread-safe depending on what the data is and how you're using it. One of them is to use thread-safe data structures like Queue from Python's queue module.
Gevent is a library based on non-blocking IO (libevent/libev) and lightweight greenlets (essentially Python coroutines). Non-blocking IO means requests waiting for network IO won't block other requests; greenlets mean we can continue to write code in synchronous style natural to Python.
The author of requests
has also created a gevent
-integration package: grequests
. Use that instead.
It supports passing in a session with the session
keyword:
import grequests
s = requests.Session()
requests = [grequests.post("http://127.0.0.1:5000",
data=json.dumps({'value': d}), session=s)
for d in range(300)]
responses = grequests.map(requests)
for r in responses:
if r.content.strip() != str(d*2):
print("Sent %s got %s" % (r.content, str(d*2)))
if r.status_code != 200:
print(r.status_code)
print(r.content)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With