Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is a Python requests Session object shared between gevent greenlets, thread-safe (between greenlets)?

Can a requests library session object be used across greenlets safely in a gevented program?

EDIT - ADDING MORE EXPLANATION:

When a greenlet yields because it has made a socket call to send the request to the server, can the same socket (inside the shared session object) be used safely by another greenlet to send its own request?

END EDIT

I attempted to test this with the code posted here - https://gist.github.com/donatello/0b399d0353cb29dc91b0 - however I got no errors or unexpected results. However, this does not validate thread safety.

In the test, I use a shared session object to make lots of requests and try to see if the object gets the requests mixed up - it is kind of naive, but I do not get any exceptions.

For convenience, I am re-pasting the code here:

client.py

import gevent
from gevent.monkey import patch_all
patch_all()

import requests
import json

s = requests.Session()

def make_request(s, d):
    r = s.post("http://127.0.0.1:5000", data=json.dumps({'value': d}))
    if r.content.strip() != str(d*2):
        print("Sent %s got %s" % (r.content, str(d*2)))
    if r.status_code != 200:
        print(r.status_code)
        print(r.content)

gevent.joinall([
    gevent.spawn(make_request, s, v)
    for v in range(300)
])

server.py

from gevent.wsgi import WSGIServer
from gevent.monkey import patch_all

patch_all()

from flask import Flask
from flask import request

import time
import json

app = Flask(__name__)

@app.route('/', methods=['POST', 'GET'])
def hello_world():
    d = json.loads(request.data)
    return str(d['value']*2)

if __name__ == '__main__':
    http_server = WSGIServer(('', 5000), app)
    http_server.serve_forever()

Exact library versions:

requirements.txt

Flask==0.10.1
Jinja2==2.7.2
MarkupSafe==0.23
Werkzeug==0.9.4
argparse==1.2.1
gevent==1.0.1
greenlet==0.4.2
gunicorn==18.0
itsdangerous==0.24
requests==2.3.0
wsgiref==0.1.2

Is there some other test that can check greenlet thread safety? The requests documentation is not too clear on this point.

like image 543
donatello Avatar asked May 29 '14 14:05

donatello


People also ask

Is Python requests session thread safe?

Unfortunately requests. Session() is not thread-safe. There are several strategies for making data accesses thread-safe depending on what the data is and how you're using it. One of them is to use thread-safe data structures like Queue from Python's queue module.

What is gevent used for?

Gevent is a library based on non-blocking IO (libevent/libev) and lightweight greenlets (essentially Python coroutines). Non-blocking IO means requests waiting for network IO won't block other requests; greenlets mean we can continue to write code in synchronous style natural to Python.


1 Answers

The author of requests has also created a gevent-integration package: grequests. Use that instead.

It supports passing in a session with the session keyword:

import grequests

s = requests.Session()

requests = [grequests.post("http://127.0.0.1:5000", 
                           data=json.dumps({'value': d}), session=s)
            for d in range(300)]

responses = grequests.map(requests)
for r in responses:
    if r.content.strip() != str(d*2):
        print("Sent %s got %s" % (r.content, str(d*2)))
    if r.status_code != 200:
        print(r.status_code)
        print(r.content)
like image 193
Martijn Pieters Avatar answered Oct 12 '22 13:10

Martijn Pieters