Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kubernetes AutoScaler Not Scaling, HPA shows target <unknown>

I'm pretty new to kubernetes, not so much with docker.

I've been working through the example but I am stuck with the autoscaler, (which doesn't seem to scale).

I am working through the example here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#step-one-run--expose-php-apache-server

You'll find the build at the bottom

kubectl create -f https://k8s.io/docs/tasks/run-application/hpa-php-apache.yaml

Which looks like this

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

kubectl get hba shows

NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         10        0          7s

Clue in the <unknown> section.

Then kubectl describe hba shows

Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 14 Apr 2018 23:05:05 +0100
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: deployments/scale.extensions "php-apache" not found
Events:
  Type     Reason          Age                From                       Message
  ----     ------          ----               ----                       -------
  Warning  FailedGetScale  12s (x14 over 6m)  horizontal-pod-autoscaler  deployments/scale.extensions "php-apache" not found

So then I manually add it in with...

kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80

Then if i run kubectl describe hpa again I get this error..

Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 14 Apr 2018 23:05:05 +0100
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetScale                1m (x21 over 11m)  horizontal-pod-autoscaler  deployments/scale.extensions "php-apache" not found
  Warning  FailedGetResourceMetric       8s (x2 over 38s)   horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  8s (x2 over 38s)   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
like image 241
Matt The Ninja Avatar asked Apr 14 '18 22:04

Matt The Ninja


People also ask

Why HPA is not working?

Hence, HPA does not respond by scaling out more replicas. In this example, the workload spike lasted for a longer period ~ 5 sec. Yet, the average CPU utilization aggregated over 30 sec = 31% < 80% targetAverageCPUUtilization. So, HPA again does not scale out the deployment.

Does HPA need metrics server?

In order to work, HPA needs a metrics server available in your cluster to scrape required metrics, such as CPU and memory utilization. One straightforward option is the Kubernetes Metrics Server.

How do you test auto scaling in Kubernetes?

To test that the horizontal pod autoscaler is working properly, you need to load test this service and make sure that the replicas increase proportionately with traffic. To do this, you will run a BusyBox pod that makes an HTTP call to your service, sleeps for 0.01 seconds, and repeats.


2 Answers

There are 2 things required to ensure your HPA is functional

  1. metrics server is running
  2. The Scalable resource (Pod/Statefulset) has CPU/Memory resource requests and limits defined. (Talking only in the scope of cpu/mem, while other custom metrics can be enabled to be used by HPA)

Check if metrics server is running -

kubectl get po -n kube-system | grep metric

If you get none listed, you don't have your metric server already running. Refer Readme and install the metrics server. Adding the CPU limits wont help unless your metrics service is running.

Now make sure you have resource requests and limits defined in your pod or your scalable target.

 resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "250m"

Also, metrics server would be useful, only while you scale based on cpu/memory.

like image 166
Satish Kumar Nadarajan Avatar answered Nov 08 '22 23:11

Satish Kumar Nadarajan


To answer the question directly, if you have set up an HPA resource and using the metrics server, and ran into this error.

This error means that in your pods, which may have more than one container, either one or both of the containers have not defined resource requests:

Resources:
     requests:
       cpu: <this is missing! Add it>

Check that all containers have resource requests defined.

To add edit and add resources directly, use kubectl edit <resource>

Kubernetes may not allow you to update if your using helm even with the force flag.

A note, this may cause downtime if you have not set PodDisruptionBudgets so set those before you ran your edit

like image 29
Margach Chris Avatar answered Nov 09 '22 00:11

Margach Chris