Is boto3 client thread-safe

Question

Is boto3 low level client for S3 thread-safe? Documentation is not explicit about it.

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#client

A similar issue is discussed in Github

https://github.com/boto/botocore/issues/1246

But still there is no answer from maintainers.

Answer 1

If you take a look at the Multithreading/Processing documentation for boto3 you can see that they recommend one client per session as there is shared data between instance that can be mutated by individual threads.

It also looks like there's an open GitHub issue for this exact question. https://github.com/boto/botocore/issues/1246

Answer 2

I recently tried using the single boto client instance using concurrent.futures.ThreadPoolExecutor. I run into exceptions coming from boto. I assume the boto client is not thread safe in this case.

The exception I got

  File "xxx/python3.7/site-packages/boto3/session.py", line 263, in client     aws_session_token=aws_session_token, config=config)   File "xxx/python3.7/site-packages/botocore/session.py", line 827, in create_client     endpoint_resolver = self._get_internal_component('endpoint_resolver')   File "xxx/python3.7/site-packages/botocore/session.py", line 694, in _get_internal_component     return self._internal_components.get_component(name)   File "xxx/python3.7/site-packages/botocore/session.py", line 906, in get_component     del self._deferred[name]