Automatically merge / rollup data in elastic search

Question

Is there an easy way to create a new index from aggregated results from another index (And maybe merge em).

I have a large index with products that are similar. They have a product ID to identify which products belong together, but they have a different URL / Price and a different title (that I want to preserve somehow in the merge so I can search it).

So if I enter 8 product lines I would love to have it all roll up into 1 product with a nested array with similar product data.

I tried the rollup API with the job below. But I couldn't get that going the way I wanted and Im getting the feeling that that is only for historical / log data. All my data has the same timestamp since I update all of this every morning.

PUT _xpack/rollup/job/product
{
  "index_pattern": "products",
  "rollup_index": "products_rollup",
  "cron": "*/30 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "field": "timestamp",
      "interval": "7d"
    },
    "terms": {
      "fields": [
        "product_id"
      ]
    }
  },
  "metrics": [
    {
      "field": "total_price",
      "metrics": [
        "min",
        "max",
        "sum"
      ]
    }
  ]
}

Thanks!

Answer 1

For now the rollup API is mainly intended to rollup numerical data in time. Not to merge documents. In your case I would merge the documents on application level and get one document with the "subdocuments" in a nested object.