input data to aws elasticsearch using boto3 or es library

Question

I have a lot of data that I want to send to aws elasticsearch. by looking at the https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-upload-data.html aws website it uses curl -Xput However I want to use python to do this therefore I've looked into boto3 documentation but cannot find a way to input data.

https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/es.html I cannot see any method that inserts data.

This seems very basic job. Any help?

Answer 1

You can send the data to elastic search using HTTP interface. Here is the code sourced from

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html

from requests_aws4auth import AWS4Auth
import boto3

host = '' # For example, my-test-domain.us-east-1.es.amazonaws.com
region = '' # e.g. us-west-1

service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

es = Elasticsearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

document = {
    "title": "Moneyball",
    "director": "Bennett Miller",
    "year": "2011"
}

es.index(index="movies", doc_type="_doc", id="5", body=document)

print(es.get(index="movies", doc_type="_doc", id="5"))

EDIT

To confirm whether data is pushed to the elastic cache under your index, you can try to do an HTTP GET by replacing the domain and index name

search-my-domain.us-west-1.es.amazonaws.com/_search?q=movies