List the memory and cpu of kubernetes nodes

Question

I am trying to list the cpu and memory usage of all the nodes in kubernetes and echo "load exceed" if the memory or cpu limit exceed some digit. I am listing the cpu and memory this command but how to apply the logic to echo that load exceeded

kubectl describe nodes | grep -A 3 "Resource .*Requests .*Limits"

Output:

Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         360m (18%)    13 (673%)
  memory                      2800Mi (84%)  9Gi (276%)
--
  Resource                    Requests          Limits
  --------                    --------          ------
  cpu                         1430m (74%)       22300m (1155%)
  memory                      2037758592 (58%)  15426805504 (441%)
--
  Resource                    Requests     Limits
  --------                    --------     ------
  cpu                         240m (12%)   5 (259%)
  memory                      692Mi (20%)  3Gi (92%)
--
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         930m (48%)    3100m (160%)
  memory                      1971Mi (59%)  3412Mi (102%)
--
  Resource                    Requests     Limits
  --------                    --------     ------
  cpu                         270m (13%)   7 (362%)
  memory                      922Mi (27%)  4Gi (122%)
--
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         530m (27%)    5 (259%)
  memory                      1360Mi (40%)  3Gi (92%)
--
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         440m (22%)    5250m (272%)
  memory                      1020Mi (30%)  3884Mi (116%)

Answer 1

Try this to extract the attributes you want by matching a regex pattern

kubectl describe nodes | grep -E -A 3 "Resource|Requests|Limits"

You can extend it like this to extract CPU or MEMORY values

grep -E -A 3 "Resource|Requests|Limits" | awk '/cpu/{print $2}'

EDIT To print if the limit is exceeded or not (example for cpu exceeding 1),

grep -E -A 3 "Limits" | awk '/cpu/{if($2 > 1) print "Limit Exceeded"; else print "Within Limits";}'

You will have to do

| awk '/memory/{print $2}' | awk -vFS="" '{print $1}'

to extract the number from memory before applying the condition as it gives the metric G with the number.

EDIT 2

This can give you the ratio based on the provided output of your grep command.

CPU

| awk '/cpu/{print $1,$2,$4}' | awk '{if($3 ~ /[0-9]$/) {print $1,$2/($3*1000)} else {print $1,$2/$3}}'

Output

cpu 0.0276923
cpu 0.0641256
cpu 0.048
cpu 0.3
cpu 0.0385714
cpu 0.106
cpu 0.0838095

MEMORY

| awk '/memory/{print $1,$2,$4}' | awk '{if($3 ~ /Gi$/) {print $1,$2/($3*1024)} else {print $1,$2/$3}}'

Output

memory 0.303819
memory 0.132092
memory 0.22526
memory 0.577667
memory 0.225098
memory 0.442708
memory 0.262616

Answer 2

As mentioned in the comments, depending on the resource type Kubernetes will behave differently and you can use kube-prometheus stack for CPU and memory exceed alerts.

• If you want to monitor your current resource consumption take a look at this page.

• I'd like to make clear that your solution concept is unfortunately flawed already because resource requests in kubernetes terms can by nature never exceed limits.

• The alternative is to use something like either the metrics server or prometheus e.g. in the form of the kube-prometheus-stack. With the latter you can easily add rules that alert you whenever an app exceeds the requests.

• The problem with limits is a little more complicated because depending on the resource type kubernetes will behave differently. If actual memory consumption exceeds the configured limit kubernetes will automatically kill the corresponding pod. On the other hand if actual cpu consumption exceeds the configured cpu limit it will lead to throttling which basically means cpu time will be capped at the configured limit.