How can I determine the current ephemeral-storage usage of a running Kubernetes pod?

Question

How can I tell with kubectl how much ephemeral storage a pod is currently using?

In a Kubernetes pod spec, I can specify resource requests and limits for CPU, memory, and ephemeral storage:

resources:
  requests:
    memory: "60Mi"
    cpu: "70m"
    ephemeral-storage: "2Gi"
  limits:
    memory: "65Mi"
    cpu: "75m"
    ephemeral-storage: "4Gi"

However, to set good requests and limits on ephemeral storage, I need to know what this value actually is for a running pod, which I can't figure out. I can get CPU and memory usage using kubectl top pod, but, from what I can tell, ephemeral storage usage is only actually calculated when making an actual eviction decision.

Answer 1

The pure raw approach for this is to use the disk usage (du) Unix command line.

Shell into your pod:

$ kubectl exec -it <pod-id> sh

Change dirs to the mount point of your ephemeral-storage (if you are using volume mounts):

$ mount # check mount points if you'd like
$ cd /mnt/of/ephemeral
$ du .

If you are not using volume mounts:

$ du .

There are other tools that you can use to get metrics:

• cAdvisor also embedded into the kubelet code, exposed under the /stats/summary or /metrics endpoint. More info here. An example output:

  $ curl -k -H 'Authorization: Bearer <Redacted>' \
  https://node-hostname:10250/stats/summary
  
  {
   "node": {
     "nodeName": "node-hostname",
     "systemContainers": [
      {
       "name": "kubelet",
      ...
      "volume": [
       {
        "time": "2018-11-08T23:52:03Z",
        "availableBytes": 1969168384,
        "capacityBytes": 1969180672,
        "usedBytes": 12288,
        "inodesFree": 480748,
        "inodes": 480757,
        "inodesUsed": 9,
        "name": "kube-proxy-token-pprwb"
       }
      ],
      "ephemeral-storage": {
       "time": "2018-11-09T00:05:10Z",
       "availableBytes": 31057477632,
       "capacityBytes": 41567858688,
       "inodesFree": 4873887,
       "inodes": 5120000
      }
  ...
  }
  

  Similarly:

  $ curl -k -H 'Authorization: Bearer <Redacted>' \
  https://node-hostname:10250/stats/summary
  
  # HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
  # TYPE apiserver_audit_event_total counter
  apiserver_audit_event_total 0
  # HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
  # TYPE apiserver_client_certificate_expiration_seconds histogram
  apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
  apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
  apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
  ...
  

  More info on kubelet authentication/authorization.

• Prometheus

More info on K8s metrics here.

Answer 2

You can do this through the raw command.

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/stats/summary"

There is also this

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/metrics/cadvisor""