Unable to publish spring boot metrics to GCP stackdriver

Question

I am trying to publish metrics of my spring boot(2.2.6.RELEASE) app running in a GCP compute engine to stackdriver. I have added the dependency of micrometer-registry-stackdriver:1.3.6 and spring-boot-actuator. Stackdriver metrics client configuration is done as given in the micrometer link. In the VM I have installed the cloud monitoring agent as mentioned here.

While running the app, I am getting the error io.grpc.StatusRuntimeException: INTERNAL: An internal error occurred.

Can anyone inform me if I am missing any other configuration?

2020-04-29 10:19:16.248  INFO 14905 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 3608
ms
2020-04-29 10:19:18.085  INFO 14905 --- [           main] i.m.s.StackdriverMeterRegistry           : publishing metrics to stackdriver every 1m
2020-04-29 10:19:20.058  INFO 14905 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-04-29 10:19:20.735  INFO 14905 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 13 endpoint(s) beneath base path '/actuator'
2020-04-29 10:19:20.897  INFO 14905 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-04-29 10:19:20.908  INFO 14905 --- [           main] com.example.demo.DemoApplication         : Started DemoApplication in 9.375 seconds (JVM running for 10.
417)
2020-04-29 10:19:27.548  INFO 14905 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-04-29 10:19:27.549  INFO 14905 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2020-04-29 10:19:27.568  INFO 14905 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 19 ms
2020-04-29 10:20:39.832  WARN 14905 --- [trics-publisher] i.m.s.StackdriverMeterRegistry           : failed to send metrics to Stackdriver

com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: An internal error occurred.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:67) ~[gax-1.48.1.jar!/:1.48.1]
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) ~[gax-grpc-1.48.1.jar!/:1.48.1]
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) ~[gax-grpc-1.48.1.jar!/:1.48.1]
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) ~[gax-grpc-1.48.1.jar!/:1.48.1]
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[api-common-1.8.1.jar!/:na]
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982) ~[guava-28.0-android.jar!/:na]
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) ~[guava-28.0-android.jar!/:na]
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) ~[guava-28.0-android.jar!/:na]
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957) ~[guava-28.0-android.jar!/:na]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) ~[guava-28.0-android.jar!/:na]
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515) ~[grpc-stub-1.23.0.jar!/:1.23.0]
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490) ~[grpc-stub-1.23.0.jar!/:1.23.0]
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.23.0.jar!/:1.23.0]
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.23.0.jar!/:1.23.0]
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.23.0.jar!/:1.23.0]
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.23.0.jar!/:1.23.0]
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.23.0.jar!/:1.23.0]
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.23.0.jar!/:1.23.0]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_252]
                at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_252]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_252]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_252]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_252]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_252]
                at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_252]
                Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57) ~[gax-1.48.1.jar!/:1.48.1]
                        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) ~[gax-1.48.1.jar!/:1.48.1]
                        at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1156) ~[google-cloud-monitoring-1.93.0.jar!/:1.93.
        0]
                        at io.micrometer.stackdriver.StackdriverMeterRegistry.publish(StackdriverMeterRegistry.java:176) ~[micrometer-registry-stackdriver-1.3.6.jar!/:1.3
        .6]
                        at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:48) [micrometer-core-1.3.6.jar!/:1.3.6]
                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_252]
                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_252]
                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_252]
                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_252]
                        ... 3 common frames omitted
        Caused by: io.grpc.StatusRuntimeException: INTERNAL: An internal error occurred.
                at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.23.0.jar!/:1.23.0]
                ... 24 common frames omitted
        2020-04-29 10:20:55.181  INFO 14905 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
         Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.

UPDATE :- I was able to publish the metrics to stackdriver after upgrading the micrometer version to 1.5.1

Answer 1

First of all, check if you have io.micrometer:micrometer-core dependency. Second, from my experience, make your meterRegistry() bean return MeterRegistry instead of StackdriverMeterRegistry.

@Bean
public MeterRegistry meterRegistry() {
    final StackdriverConfig stackdriverConfig = new StackdriverConfig() {
        @Override
        public String projectId() {
            return "your-project-name";
        }

        @Override
        public String get(final String key) {
            return null; //this field is ignored
        }
    };
    return StackdriverMeterRegistry.builder(stackdriverConfig).build();
}

Also, I hope you have management.* properties properly configured. This is my example

management:
  endpoints:
    metrics:
      enabled: true
    prometheus:
      enabled: true
    web:
      exposure:
        include: '*'
  metrics:
    tags:
      application: your-app
    export:
      prometheus:
        enabled: true
    stackdriver:
      enabled: true
      step: 1m

Answer 2

I also got this error and for me the following configuration fixed the issue for me:

management:
  metrics:
    tags:
      app: my-unique-service-label
      instance: ${random.uuid}

If your application runs inside a pod/multiple times, you might want to add the pod name/instance id to the labels to keep them distinct.

See also:

• How to define additional or custom tags for default Spring Boot 2 metrics?
• https://github.com/GoogleCloudPlatform/spring-cloud-gcp/issues/1223
• https://github.com/micrometer-metrics/micrometer/issues/1268