Django haystack with elasticsearch, indexing issue

Question

Im using django-haystack with elasticsearch but there is a problem with indexing. When rebuilding my index

 python manage.py rebuild_index 

following error is raised:

Traceback (most recent call last):
  File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 210, in handle_label
    self.update_backend(label, using)
  File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 256, in update_backend
    do_update(backend, index, qs, start, end, total, self.verbosity)
  File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 78, in do_update
    backend.update(index, current_qs)
  File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 177, in update
    self.conn.bulk_index(self.index_name, 'modelresult', prepped_docs, id_field=ID)
  File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 95, in decorate
    return func(*args, query_params=query_params, **kwargs)
  File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 366, in bulk_index
    query_params=query_params)
  File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 221, in send_request
    **({'data': request_body} if body else {}))
  File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 387, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 345, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 448, in send
    r = adapter.send(request, **kwargs)
  File "/home/palo/.virtualenvs/toro/src/requests/requests/adapters.py", line 324, in send
    raise Timeout(e)
Timeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Request timed out. (timeout=10)
Timeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Request timed out. (timeout=10)

I used django-haystack - 2.0.0-beta, pyelasticsearch - 0.5, elasticsearch 0.20.6, java version "1.6.0_24"

Haystack Settings

    HAYSTACK_CONNECTIONS = {
        'default': {
            'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
            'URL': 'http://127.0.0.1:9200/',
            'INDEX_NAME': 'haystack',
        },
    }

And Im sure my elasticsearch serivce is running.

Answer 1

This does not necessarily mean that your es server is down, especially if you get something reasonable returned with curl -I "127.0.0.1:9200". More likely, this is an issue of your request simply not getting enough time given the speed of connections involved.

Interestingly, the default timeout set in pyelasticsearch is 60 seconds, see def __init__(self, urls, timeout=60, max_retries=0, revival_delay=300): in https://github.com/rhec/pyelasticsearch/blob/master/pyelasticsearch/client.py. However, haystack overwrites that with its default setting, which is 10 seconds, as per self.timeout = connection_options.get('TIMEOUT', 10) in https://github.com/toastdriven/django-haystack/blob/master/haystack/backends/__init__.py.

As you can see though, haystack allows you to easily modify your setting, by adding 'TIMEOUT': 60, to your engine configuration.

And solved :)