Connection Pooling with NEST ElasticSearch Library

Question

I'm currently using the NEST ElasticSearch C# Library for interacting with ElasticSearch. My project is an MVC 4 WebAPI project that basically builds a RESTful webservice for accessing directory assistance information.

We've only just started working with NEST, and have been stumbling over the lack of documentation. What's there is useful, but it's got some very large holes. Currently, everything we need works, however, we're running into an issue with connections sometimes taking up to a full second. What we'd like to do is use some sort of connection pooling, similar to how you'd interact with SQL Server.

Here is the documentation on how to connect using nest: http://mpdreamz.github.com/NEST/concepts/connecting.html

Here is the relevant code snippet from our project:

public class EOCategoryProvider : IProvider
{
    public DNList ExecuteQuery(Query query)
    {
        //Configure the elastic client and it's settings
        ConnectionSettings elasticSettings = new ConnectionSettings(Config.server, Config.port).SetDefaultIndex(Config.index);
        ElasticClient client = new ElasticClient(elasticSettings);

        //Connect to Elastic
        ConnectionStatus connectionStatus;
        if (client.TryConnect(out connectionStatus))
        {
            // Elastic Search Code here ...
        } // end if
    } // end ExecuteQuery
} // end EOCategoryProvider

From looking at the documentation, I can't see any provisions for a connection pool. I've been thinking about implementing my own (having, say 3 or 4 ElasticClient objects stored, and selecting them round-robin style), but I was wondering if anyone had a better solution. If not, does anyone have advice on the best way to implement a connection pool by hand? Any articles to point to?

Thanks for anything you guys come up with.

Update: This seems to have been related to calling TryConnect on every request, and the particular network setup. The problem completely disappeared when using a machine on the same network as the Elastic box; My development machine (which averages 350ms to the Elastic box) seemed to fail to make http connections sometimes, which caused the long times in TryConnect.

Answer 1

You don't have to call TryConnect() each time you do a call to Elasticsearch. It's basically a sanity check call for when your application starts.

NEST is the C# REST client for Elasticsearch and the default IConnection uses WebRequest.Create which already pools TCP connections.

Review the actual implementation: https://github.com/elastic/elasticsearch-net/blob/master/src/Elasticsearch.Net/Connection/HttpConnection.cs

Reusing ElasticClient won't offer any performance gains since each call already gets its own HttpWebRequest. The whole client is built stateless on purpose.

I am however very interested in why calls are taking 1 second for you. Could you post the actual NEST code, how you are are measuring the calls and describe your data.

Disclaimer: I'm the author of NEST.