Django-Haystack using Amazon Elasticsearch hosting with IAM credentials

Question

I am hoping to use Amazon's Elasticsearch server to power a search of longtext fields in a Django database. However, I also don't want to expose this search to those who don't have a log in and don't want to rely on security through obscurity or some IP restriction tactic (unless it would work well with an existing heroku app, where the Django app is deployed).

Haystack seems to go a long way toward this, but there doesn't seem to be an easy way to configure it to use Amazon's IAM credentials to access the Elasticsearch service. This functionality does exist in elasticsearch-py, whichi it uses.

https://elasticsearch-py.readthedocs.org/en/master/#running-with-aws-elasticsearch-service

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

host = 'YOURHOST.us-east-1.es.amazonaws.com'
awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es')

es = Elasticsearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)
print(es.info())

Regarding using HTTP authorization, I found this under issues at https://github.com/django-haystack/django-haystack/issues/1046

from urlparse import urlparse
parsed = urlparse('https://user:pass@host:port')
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': parsed.hostname,
        'INDEX_NAME': 'haystack',
        'KWARGS': {
            'port': parsed.port,
            'http_auth': (parsed.username, parsed.password),
            'use_ssl': True,
        }
    }
}

I am wondering if there is a way to combine these two, something like the following (which, as expected, gives an error since it's more than just a user name and password):

from requests_aws4auth import AWS4Auth
awsauth = AWS4Auth([AACCESS_KEY],[SECRET_KEY],[REGION],'es')


HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': [AWSHOST],
        'INDEX_NAME': 'haystack',
        'KWARGS': {
            'port': 443,
            'http_auth': awsauth,
            'use_ssl': True,
            'verify_certs': True
        }
    },
}

The error here:

TypeError at /admin/
must be convertible to a buffer, not AWS4Auth

Request Method:     GET
Request URL:    http://127.0.0.1:8000/admin/
Django Version:     1.7.7
Exception Type:     TypeError
Exception Value:    

must be convertible to a buffer, not AWS4Auth

Exception Location:     /usr/lib/python2.7/base64.py in b64encode, line 53

Any ideas on how to accomplish this?

Answer 1

You are one step from success, add connection_class to KWARGS and everything should work as expected.

import elasticsearch

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': [AWSHOST],
        'INDEX_NAME': 'haystack',
        'KWARGS': {
            'port': 443,
            'http_auth': awsauth,
            'use_ssl': True,
            'verify_certs': True,
            'connection_class': elasticsearch.RequestsHttpConnection,
        }
    },
}