Menu
Python's popular Requests library is said to be thread-safe on its home page, but no further details are given. If I call requests.session(), can I then safely pass this object to multiple threads like so:
session = requests.session() for i in xrange(thread_count): threading.Thread( target=target, args=(session,), kwargs={} ) and make requests using the same connection pool in multiple threads?
If so, is this the recommended approach, or should each thread be given its own connection pool? (Assuming the total size of all the individual connection pools summed to the size of what would be one big connection pool, like the one above.) What are the pros and cons of each approach?
869
A requests Session is documented as threadsafe but there are still a couple corner cases where it isn't perfectly threadsafe. The best way to use a Session is to use one per thread. The implementation provided by the toolbelt is naïve.
Threading in Python is simple. It allows you to manage concurrent threads doing work at the same time. The library is called “threading“, you create “Thread” objects, and they run target functions for you. You can start potentially hundreds of threads that will operate in parallel.
No, it is not thread safe - the two threads are essentially modifying the same variable simultaneously.
Like the other have said, Python objects are mostly thread-safe. Although you will need to use Locks in order to protect an object in a place that require it to go through multiple changes before being usable again.
After reviewing the source of requests.session, I'm going to say the session object might be thread-safe, depending on the implementation of CookieJar being used.
Session.prepare_request reads from self.cookies, and Session.send calls extract_cookies_to_jar(self.cookies, ...), and that calls jar.extract_cookies(...) (jar being self.cookies in this case).
The source for Python 2.7's cookielib acquires a lock (threading.RLock) while it updates the jar, so it appears to be thread-safe. On the other hand, the documentation for cookielib says nothing about thread-safety, so maybe this feature should not be depended on?
UPDATE
If your threads are mutating any attributes of the session object such as headers, proxies, stream, etc. or calling the mount method or using the session with the with statement, etc. then it is not thread-safe.
99
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
