How to get list of metrics available for HPA?

Question

I have GCP cluster which contains GKE application:

I want to scale application using HPA

Based on supporting for metrics

HPA able to read metrics from

• metrics.k8s.io (resource metrics)

• custom.metrics.k8s.io(custom metrics)

• external.metrics.k8s.io(external metrics)

How could I check what metrics available? How could try this API on my own ? Is it possible at all?

P.S.

Based on suggested answer I executed command:

 kubectl get --raw https://MY-KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods

Response is:

{
  "items": [
    {
      "metadata": {
        "name": "prometheus-adapter-69fcdd56bc-2plh7",
        "namespace": "default",
        "selfLink": "/\r\napis/metrics.k8s.io/v1beta1/namespaces/default/pods/prometheus-adapter-69fcdd56bc-2plh7",
        "creationTimestamp": "2020-02-05T10:56:02Z"
      },
      "timestamp": "2020-02-05T10:55:22Z",
      "window": "30s",
      "containers": [
        {
          "name": "prometheus-adapter",
          "usage": {
            "cpu": "15\r\n31939n",
            "memory": "10408Ki"
          }
        }
      ]
    },
    {
      "metadata": {
        "name": "stackdriver-exporter-76fdbc9d8f-c285l",
        "namespace": "default",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/stackdriver-exporter-76fdbc9d8f-c285l",
        "creationTimestamp": "2020-0\r\n2-05T10:56:02Z"
      },
      "timestamp": "2020-02-05T10:55:22Z",
      "window": "30s",
      "containers": [
        {
          "name": "stackdriver-exporter",
          "usage": {
            "cpu": "79340n",
            "memory": "2000Ki"
          }
        }
      ]
    }
  ],
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"
  }
}


$ kubectl top pods
NAME                                    CPU(cores)   MEMORY(bytes)
prometheus-adapter-69fcdd56bc-2plh7     2m           10Mi
stackdriver-exporter-76fdbc9d8f-c285l   1m           1Mi

But I still don't see all metrics available for HPA

Answer 1

Metrics server exposes metrics via below APIs.

1. /nodes - all node metrics; type []NodeMetrics
2. /nodes/{node} - metrics for a specified node; type NodeMetrics
3. /namespaces/{namespace}/pods - all pod metrics within namespace with support for all-namespaces; type []PodMetrics
4. /namespaces/{namespace}/pods/{pod} - metrics for a specified pod; type PodMetrics

You can view available metrics as below for example

$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nodes",
      "singularName": "",
      "namespaced": false,
      "kind": "NodeMetrics",
      "verbs": [
        "get",
        "list"
      ]
    },
    {
      "name": "pods",
      "singularName": "",
      "namespaced": true,
      "kind": "PodMetrics",
      "verbs": [
        "get",
        "list"
      ]
    }
  ]
}

$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods
{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"
  },
  "items": []
}

$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods
{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods"
  },
  "items": [
    {
      "metadata": {
        "name": "coredns-bcccf59f-jfl6x",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-bcccf59f-jfl6x",
        "creationTimestamp": "2021-02-17T20:31:29Z"
      },
      "timestamp": "2021-02-17T20:30:27Z",
      "window": "30s",
      "containers": [
        {
          "name": "coredns",
          "usage": {
            "cpu": "1891053n",
            "memory": "8036Ki"
          }
        }
      ]
    },
    {
      "metadata": {
        "name": "coredns-bcccf59f-vmfvv",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-bcccf59f-vmfvv",
        "creationTimestamp": "2021-02-17T20:31:29Z"
      },
      "timestamp": "2021-02-17T20:30:25Z",
      "window": "30s",
      "containers": [
        {
          "name": "coredns",
          "usage": {
            "cpu": "1869226n",
            "memory": "8096Ki"
          }
        }
      ]
    }
  ]
}

You can also use command kubectl top pods which internally calls the above API.

Custom Metrics

These are provided by adapters developed by vendors and what metrics are available will depend on the adapter. Once you know the metrics name You can use API to access it.

You can view available metrics as below and get the metrics name.

kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/custom.metrics.k8s.io/v1beta1

External Metrics

These are provided by adapters developed by vendors and what metrics are available will depend on the adapter. Once you know the metrics name You can use API to access it.

You can view available metrics as below and get the metrics name.

kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/external.metrics.k8s.io/v1beta1

Edit:

You already have Prometheus adapter but if the metric is not exposed as custom metrics to be consumable by HPA then you need to expose the required metrics. Refer to this guide for this.

Answer 2

On GKE case is bit different.

As default Kubernetes have some built-in metrics (CPU and Memory). If you want to use HPA based on this metric you will not have any issues.

In GCP concept:

• Custom Metrics are used when you want to use metrics exported by Kubernetes workload or metric attached to Kubernetes object such as Pod or Node.
• External Metrics- Metrics sent to Workspaces with a metric type beginning external.googleapis.com are known as external metrics. The metrics are typically exported by open-source projects and third-party providers. More details can be found here. Stackdriver Monitoring treats external metrics the same as custom metrics, with one exception. For external metrics, a resource_type of global is invalid and results in the metric data being discarded.

As GKE is integrated with Stackdriver

Google Kubernetes Engine (GKE) includes native integration with Stackdriver Monitoring and Stackdriver Logging. When you create a GKE cluster, Stackdriver Kubernetes Engine Monitoring is enabled by default and provides a monitoring dashboard specifically tailored for Kubernetes.

With Stackdriver Kubernetes Engine Monitoring, you can control whether or not Stackdriver Logging collects application logs. You also have the option to disable the Stackdriver Monitoring and Stackdriver Logging integration altogether.

Check Available Metrics

As you are using cloud environment - GKE, you can find all default available metrics by curiling localhost on proper port. You have to SSH to one of Nodes and then curl metric-server $ curl localhost:10255/metrics.

Second way is to check available metrics documentation.

IMPORTANT

You can see available metrics, however to use them in HPA you need deploy Adapters like Stackdriver adapter or Prometheus adapter. In default (version 1.13.11) GKE cluster you have already deployed metrics-server, heapster(deprecated in newer versions) and prometheus-to-sd-XXX. If you would like to use Stackdriver, you would have many config already applied, but if you want to use Prometheus you will need adjust Prometheus operators, adapters, deployments. Details can be found here.

On GKE docs, you can find some tutorials to use HPA with Custom Metrics or with HPA with External Metrics. You can also read about GKE monitoring with Prometheus and Stackdriver, depends on your needs.

As GKE is integrated with Stackdriver you can read article about Enabling Monitoring.