Ways to only process new(index after last run) data in Elasticsearch?

Question

Is there a way to get the date and time that an elastic search document was written?

I am running es queries via spark and would prefer NOT to look through all documents that I have already processed. Instead I would like read the only documents that were ingested between the last time the program ran and now.

What is the best most efficient way to do this?

I have looked at;

• updating to add a field with an array with booleans for if its been looked at by which analytic. The negative is waiting for the update to occur.
• index per time frame method, which would be to break down the current indexes into smaller ones so by hour.The negative I see is the number of open file descriptors.
• ??

Elasticsearch version 5.6

Answer 1

I posted the question on the elasticsearch discussion board and it appears using the ingest pipeline is the best option.