Elastic search - one index vs multiple indexes?

Question

I'm working on a solution to store application logs in Elastic Search for many applications across many development teams. The structure of each log entry is identical with an "app" field to indicate the application.

The #1 goal is to support efficient querying within a single "app". Querying across all apps, while still important, would be secondary.

I'm trying to determine what is best:

EDIT: in both cases I will use time-based indexes.

multiple index series

Each "app" would have a series of time-based indexes (app1-2017-04-01,app1-2017-04-02,... etc.) The user would perform searches directly against these smaller indexes. The thought here is that since the indexes are smaller in size, maybe querying against them is faster?

single index series

Use one giant index series to represent all application logs (e.g. logs-2017-04-01, logs-2017-04-02, ... etc) Users would query the "app" field to narrow their search results.

Which is faster in this case? I'm curious about the overhead cost of additional indexes

Answer 1

In most cases multiple indexes are better:

1. Searching against smaller dataset is faster
2. You are less limited in mapping structure. If you need to change it for new data, you can keep old data without reindexing and just put new mapping for new index
3. It's more scalable and flexible. You can keep old indexes on a different hard drive or a different machine.
4. You still can search against multiple indexes, if required.
5. The overhead for index is small. If you have lots of documents per index, documents take much more space than index metadata. If not, you can take a smaller time period to split your log indexes