Writing to custom timeseries with the Google Cloud Monitoring v3 api

Question

Google has warned that the v2 monitoring api is now deprecated and will be going away soon. However, migrating to v3 is proving to be a bit difficult. I'm trying to write a custom metric and am getting the following error responses:

Services > Google Monitoring API v3 > monitoring.projects.timeSeries.create

{
    "timeSeries": [{
        "metric": {
            "type": "custom.googleapis.com/test_metric",
            "labels": {
                "payment_type": "Paypal"
            }
        },
        "resource": {
            "type": "custom.googleapis.com/test_metric",
            "labels": {
                "payment_type": "Paypal"
            }
        },
        "metricKind": "GAUGE",
        "valueType": "INT64",
        "points": [{
            "interval": {
                "endTime": "2016-03-20T15:01:23.045123456Z",
                "startTime": "2016-03-20T15:01:23.045123456Z"
            },
            "value": {
                "int64Value": "2"
            }
        }]
    }]
}

{
  "error": {
  "code": 400,
  "message": "Field timeSeries[0].resource.type had an invalid value of \"custom.googleapis.com/test_metric\": Unrecognized resource name.",
  "status": "INVALID_ARGUMENT"
}

The "resource" field is required, and docs say it's the "MonitoredResource"... but I don't see any api for creating one, only for listing. Taking a wild guess and setting it to "global" seems to get me a bit further and gives me this different error:

{
 "error": {
  "code": 400,
  "message": "Field timeSeries[0].resource.labels[0] had an invalid value of \"payment_type\": Unrecognized resource label.",
  "status": "INVALID_ARGUMENT"
 }
}

Listing the metric descriptors shows that payment_type exists:

Services > Google Monitoring API v3 > monitoring.projects.metricDescriptors.list

{
 "name": "projects/gearlaunch-hub-sandbox/metricDescriptors/custom.googleapis.com/test_metric",
 "labels": [
  {
   "key": "payment_type"
  }
 ],
 "metricKind": "GAUGE",
 "valueType": "INT64",
 "description": "Test",
 "type": "custom.googleapis.com/test_metric"
}

I've read through the migration guides and related docs, but am still stymied. Anyone know what I'm missing here?

Update: While it looks to be possible to get this working by removing "resource.labels" from the json, I'm still looking for a way to get this working via the java client api.

Update 2: The accepted (self answered) question shows how to do this with the java api.

Answer 1

Looks like the answer is to use "resource" of "type": "global" but leave off "labels":

{
    "timeSeries": [{
        "metric": {
            "type": "custom.googleapis.com/test_metric",
            "labels": {
                "payment_type": "Paypal"
            }
        },
        "resource": {
            "type": "global"
        },
        "metricKind": "GAUGE",
        "valueType": "INT64",
        "points": [{
            "interval": {
                "endTime": "2016-03-23T01:01:23.045123456Z",
                "startTime": "2016-03-23T01:01:23.045123456Z"
            },
            "value": {
                "int64Value": "2"
            }
        }]
    }]
}

This gives me a 200 OK response and adds the data to the time series.

This works from the api explorer directly. The equivalent code using java client api is:

public String writeCustomMetricValue(final String name, final Map<String, String> labels, final Long value) {
    Preconditions.checkNotNull(name);
    Preconditions.checkNotNull(labels);
    Preconditions.checkNotNull(value);

    final String now = DateTime.now().withZone(DateTimeZone.UTC).toString();

    final TimeInterval interval = new TimeInterval();
    interval.setStartTime(now);
    interval.setEndTime(now);

    final TypedValue pointValue = new TypedValue();
    pointValue.setInt64Value(value);

    final Point point = new Point();
    point.setInterval(interval);
    point.setValue(pointValue);

    final MonitoredResource resource = new MonitoredResource();
    resource.setType("global");

    final Metric metric = new Metric();
    metric.setType("custom.googleapis.com/" + name);

    final TimeSeries series = new TimeSeries();
    series.setMetric(metric);
    series.setPoints(Arrays.asList(point));
    series.setResource(resource);
    series.setMetricKind("GAUGE");

    final List<TimeSeries> timeseries = new ArrayList<>();
    timeseries.add(series);

    final CreateTimeSeriesRequest content = new CreateTimeSeriesRequest();
    content.setTimeSeries(timeseries);

    metric.setLabels(labels);

    try {
        return service().projects().timeSeries().create("projects/" + env.getProjectId().getId(), content).execute().toPrettyString();

    } catch (Exception e) {
        throw new RuntimeException("Name=" + name + ", labels=" + labels + ", value=" + value, e);
    }
}

Using:

<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-monitoring</artifactId>
    <version>v3-rev3-1.21.0</version>
</dependency>

Answer 2

When writing a data point, you must specify both the Metric and MonitoredResource to identify a unique time series. The |global| MonitoredResource has no labels, so it is already fully specified. It appears that your custom metric of type |custom.googleapis.com/test_metric| has a label named |payment_type|. In order to fully specify the Metric, you must assign a value to the |payment_type| field.

Try that and let me know how it works.