Is there a Python ElasticSearch client that supports asynchronous requests?

Question

I'm looking for an ElasticSearch Python client that can make asynchronous requests. For example, I'd like to write this code,

query1_future = es.search('/foobar', query1_json)
query2_future = es.search('/baz', query2_json) # Submit query 2 right after query 1, don't wait for its response
query1 = query1_future.get()
query2 = query2_future.get()

However, I don't see any clients (PyES, or the official client, for example) supporting this. Further, the two I'm familiar with couple the request logic with the response processing logic, so modifying them myself seems difficult. Perhaps a sufficient interim solution would be to use the asynchronous version of Requests, grequests?

Also, it's worth pointing out that ElasticSearch's _msearch may be a better-performing option, but for real-world applications it'd require some code restructuring.

Answer 1

Just came across this question. There is an official asynchronous Elasticsearch client based on asyncio:

https://github.com/elastic/elasticsearch-py-async

Answer 2

You can also consider the following options to perform I/O without blocking main executing process using existent clients:

• Use multithreading on Jython or IronPython (they do not have GIL and take advantage of multiple CPU cores)
• Use ProcessPoolExecutor on Python3
• Use gevent with sockets monkey pathching to force existent clients work with gevent sockets that actually makes the client asynchronous but also request some additional code to manage results

Gevent usage is the most lightweight (for RAM / CPU resources) and allows processing of the most intensive I/O, but it's also the most complex among the listed solutions. Also note that it works in the single process and to use advantage of multiple cores multiprocessing package should be used.