Kubernetes AutoScaler Not Scaling, HPA shows target <unknown>

Question

I'm pretty new to kubernetes, not so much with docker.

I've been working through the example but I am stuck with the autoscaler, (which doesn't seem to scale).

I am working through the example here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#step-one-run--expose-php-apache-server

You'll find the build at the bottom

kubectl create -f https://k8s.io/docs/tasks/run-application/hpa-php-apache.yaml

Which looks like this

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

kubectl get hba shows

NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         10        0          7s

Clue in the <unknown> section.

Then kubectl describe hba shows

Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 14 Apr 2018 23:05:05 +0100
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: deployments/scale.extensions "php-apache" not found
Events:
  Type     Reason          Age                From                       Message
  ----     ------          ----               ----                       -------
  Warning  FailedGetScale  12s (x14 over 6m)  horizontal-pod-autoscaler  deployments/scale.extensions "php-apache" not found

So then I manually add it in with...

kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80

Then if i run kubectl describe hpa again I get this error..

Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 14 Apr 2018 23:05:05 +0100
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetScale                1m (x21 over 11m)  horizontal-pod-autoscaler  deployments/scale.extensions "php-apache" not found
  Warning  FailedGetResourceMetric       8s (x2 over 38s)   horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  8s (x2 over 38s)   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)

Answer 1

There are 2 things required to ensure your HPA is functional

1. metrics server is running
2. The Scalable resource (Pod/Statefulset) has CPU/Memory resource requests and limits defined. (Talking only in the scope of cpu/mem, while other custom metrics can be enabled to be used by HPA)

Check if metrics server is running -

kubectl get po -n kube-system | grep metric

If you get none listed, you don't have your metric server already running. Refer Readme and install the metrics server. Adding the CPU limits wont help unless your metrics service is running.

Now make sure you have resource requests and limits defined in your pod or your scalable target.

 resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "250m"

Also, metrics server would be useful, only while you scale based on cpu/memory.

Answer 2

To answer the question directly, if you have set up an HPA resource and using the metrics server, and ran into this error.

This error means that in your pods, which may have more than one container, either one or both of the containers have not defined resource requests:

Resources:
     requests:
       cpu: <this is missing! Add it>

Check that all containers have resource requests defined.

To add edit and add resources directly, use kubectl edit <resource>

Kubernetes may not allow you to update if your using helm even with the force flag.

A note, this may cause downtime if you have not set PodDisruptionBudgets so set those before you ran your edit