Spring boot with micrometer Elasticsearch registry indexes only empty documents

Question

I have a simple spring boot 2.1.7.RELEASE project using micrometer Elasticsearch registry (using Elasticsearch 7.2.0). The project is available here on github. It has only two classes and looks like this

pom.xml has the following dependencies:

<dependencies>
  <dependency>
    <artifactId>spring-boot-starter-web</artifactId>
    <groupId>org.springframework.boot</groupId>
  </dependency>

  <dependency>
    <artifactId>spring-boot-starter-actuator</artifactId>
    <groupId>org.springframework.boot</groupId>
  </dependency>

  <dependency>
    <artifactId>micrometer-registry-elastic</artifactId>
    <groupId>io.micrometer</groupId>
  </dependency>
</dependencies>

and two classes: MicrometerElasticApplication:

@SpringBootApplication
public class MicrometerElasticApplication {
  public static void main(final String[] args) {
    SpringApplication.run(MicrometerElasticApplication.class, args);
  }
}

and TestController

@RestController
public class TestController {
  @ResponseStatus(HttpStatus.OK)
  @GetMapping("/test")
  public void sendMessage() {
    System.out.println("Received a test message");
  }
}

Once I start the app, I can see in the logs

i.m.elastic.ElasticMeterRegistry         : publishing metrics to elastic every 1m

which means metrics are sent, but then when I check what is indexed in the Elasticsearch, I can only see documents like this one

{
    "_index": "metrics-2019-08",
    "_type": "_doc",
    "_id": "nWuMdWwBxBoi4XILEHVK",
    "_score": 1.0
}

so no fields, just document metadata. Even after hitting the /test endpoint server times, nothing changes in the metrics index.

My understanding from reading the official documentation here and checking the common properties here is that Spring by default is going to add metrics for JVM, CPU... and even measure timings for all MVC requests. Right now, I'm not getting anything of those, just empty documents. Setting the property management.metrics.web.server.auto-time-requests to true does not change anything.

Anyone see what I'm missing here?

UPDATE

I put a break point to ElasticMeterRegistry.publish method and the request sent to Elaticsearch /_bulk API looks good to me

POST http://localhost:9200/metrics-2019-08/_bulk

{ "index" : {} }
{"@timestamp":"2019-08-09T10:49:18.826Z","name":"jvm_memory_max","type":"gauge","area":"heap","id":"PS Survivor Space","value":1.5204352E7}
{ "index" : {} }
{"@timestamp":"2019-08-09T10:49:18.826Z","name":"jvm_threads_states","type":"gauge","state":"terminated","value":0.0}
...

When I send this request using Postman, all documents are saved as empty docs, although Elasticsearch does not report any errors, "errors": false in the response

{
    "took": 8,
    "errors": false,
    "items": [
        ...
    ]
}

Answer 1

The index metrics-2019-08 create by the ElasticMeterRegistry has in its mapping set _source to false

GET http://localhost:9200/metrics-2019-08/_mapping

Response is

"metrics-2019-08": {
    "mappings": {
        "_source": {
            "enabled": false
        }
        ...
    }
}

From the Elasticserch documentation here

_source field itself is not indexed (and thus is not searchable), but it is stored so that it can be returned when executing fetch requests, like get or search.

So if _source is false, then every request to get documents returns empty source. The rational behind disabling this is documents are only going to be used for aggregations (avg, min, max, sum...), _source is not needed for those, so to save disc space _source is not stored.

To disable this behavior set management.metrics.export.elastic.auto-create-index to false. If you already run it with true, you need to delete the existing template with

DELETE http://localhost:9200/_template/metrics_template

After that, create the template for metrics indices like this:

POST http://localhost:9200/_template/metrics_template
{
  "index_patterns": [
    "metrics*"
  ],
  "mappings": {
    "_source": {
      "enabled": true
    },
    "properties": {
      "name": {
        "type": "keyword"
      },
      "count": {
        "type": "double"
      },
      "value": {
        "type": "double"
      },
      "sum": {
        "type": "double"
      },
      "mean": {
        "type": "double"
      },
      "duration": {
        "type": "double"
      },
      "max": {
        "type": "double"
      },
      "total": {
        "type": "double"
      },
      "unknown": {
        "type": "double"
      },
      "active": {
        "type": "double"
      }
    }
  }
} 

This template is the same as one used by micrometer, apart from _source which is set to true. After this, the documents will appear in fetch requests, like get or search.