Kubernetes HPA deployment cannot find target resource

Question

I'm trying to deploy my own HPA without a success. Although while trying to deploy the official PHP of kubernetes, it worked as planned. When I tryied to deploy my own Test Deployment with the HPA, it didn't work.

Compare 2 HPA deployments processes - Kubernetes official deployment VS My Test deployment:

Deploy the Official Kubernetes template image:

$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
service "php-apache" created
deployment "php-apache" created

My own Test Deployment result

{
   "apiVersion": "autoscaling/v1",
   "kind": "HorizontalPodAutoscaler",
   "metadata": {
       "annotations": {
           "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"False\",\"lastTransitionTime\":\"2019-12-22T20:39:59Z\",\"reason\":\"FailedGetScale\",\"message\":\"the HPA controller was unable to get the target's current scale: deployments/scale.apps \\\"gw-autoscale-t6\\\" not found\"}]"
       },
       "creationTimestamp": "2019-12-22T20:39:44Z",
       "labels": {
           "app": "gw-autoscale-t6"
       },
       "name": "gw-autoscale-t6",
       "namespace": "dev",
       "resourceVersion": "17299134",
       "selfLink": "/apis/autoscaling/v1/namespaces/dev/horizontalpodautoscalers/gw-autoscale-t6",
       "uid": "2f7e014c-24fb-11ea-a4d8-a28620329da6"
   },
   "spec": {
       "maxReplicas": 3,
       "minReplicas": 1,
       "scaleTargetRef": {
           "apiVersion": "apps/v1",
           "kind": "Deployment",
           "name": "gw-autoscale-t6"
       },
       "targetCPUUtilizationPercentage": 80
   },
   "status": {
       "currentReplicas": 0,
       "desiredReplicas": 0
   }
}

The HPA deployment yaml I used for both of the deployment above (Official Deployment and my Test Deployment):

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.spec.replicaCountMin }}
  maxReplicas: {{ .Values.spec.replicaCountMax }}
  targetCPUUtilizationPercentage: 50
  selector:
    matchLabels:
      app: {{ .Values.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
        release: {{ .Release.Name }}
        heritage: {{ .Release.Service }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

The results:

PHP HPA is working as planned

NAME        REFERENCE              TARGETS  MINPODS   MAXPODS   REPLICAS   AGE
php-apache  Deployment/php-apache  0%/50%    1        3         1          3d7h

Test Deployment HPA is not working

NAME              REFERENCE                    TARGETS         MINPODS  MAXPODS   REPLICAS   AGE
gw-autoscale-t6   Deployment/gw-autoscale-t6   <unknown>/80%   1        3         0          11m

The Test Deployment HPA error

Name:                                                  gw-autoscale-t6
Namespace:                                             dev
Labels:                                                app=gw-autoscale-t6
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 22 Dec 2019 22:39:44 +0200
Reference:                                             Deployment/gw-autoscale-t6
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 80%
Min replicas:                                          1
Max replicas:                                          3
Deployment pods:                                       0 current / 0 desired
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: deployments/scale.apps "gw-autoscale-t6" not found
Events:
  Type     Reason          Age                 From                       Message
  ----     ------          ----                ----                       -------
  Warning  FailedGetScale  27s (x81 over 20m)  horizontal-pod-autoscaler  deployments/scale.apps "gw-autoscale-t6" not found

• I have also tried a lot of other types of deployment files.

• My Metrics-Service is installed.

• I'm installing my deployment with Helm.

What can I do to solve this one?

---

The solution I found was to add the 'resources' property to the Test Deployment yaml file:

For instance: in case of scale by cpu usage, use it as followed in the deployment file.

resources:
    requests:
       cpu: {{ .Values.request.cpu }}
    limits:
       cpu: {{ .Values.limits.cpu }}

Answer 1

The way I solved it is by adding the cpu resources to the deployment's file and keeping only the necessary HPA deployment yaml fields.

HPA deployment file

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.spec.replicaCountMin }} 
  maxReplicas: {{ .Values.spec.replicaCountMax }} 
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.spec.cpuUtil }} 

Deployment file - resource addition

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
        release: {{ .Release.Name }}
        heritage: {{ .Release.Service }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        resources:
          requests:
            cpu: {{ .Values.request.cpu }}
          limits:
            cpu: {{ .Values.limits.cpu }} 

You must add this code:

resources:
  requests:
     cpu: {{ .Values.request.cpu }}
  limits:
     cpu: {{ .Values.limits.cpu }}

Hope it helps :)

Answer 2

It looks like something is wrong with the deployment.

I have tried to reproduce the situation and it looks like you are creating autoscaler from file with kubectl create -f hpa.yaml that looks for the Deployment/gw-autoscale-t6 that is not present in cluster.

That is why you are getting the following error:

the HPA controller was unable to get the target''s current scale: deployments/scale.apps\ "gw-autoscale-t6" not found"

for my tests I've created a hpa.yaml, applied it and looked for the situation:

$ date && kubectl get hpa && kubectl get deployments php-apache 

Tue 31 Dec 2019 10:59:37 AM CET

NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         10        0          10m

Error from server (NotFound): deployments.extensions "php-apache" not found

it gave me <unknown> as a TARGETS value. And Zero as REPLICAS

As you can see, the Deployment itself (the one we are trying to scale with hpa.yaml) doesn't exist yet.

Upon creation of a deployment:

$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80

service/php-apache created
deployment.apps/php-apache created

I have waited for a minute to let HPA to warm-up and calculate everything:

$ kubectl get hpa -o yaml

apiVersion: v1
items:
- apiVersion: autoscaling/v1
  kind: HorizontalPodAutoscaler
  metadata:
    annotations:
      autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2019-12-31T09:59:47Z","reason":"ScaleDownStabilized","message":"recent
        recommendations were higher than current one, applying the highest recent
        recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2019-12-31T10:00:33Z","reason":"ValidMetricFound","message":"the
        HPA was able to successfully calculate a replica count from cpu resource utilization
        (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2019-12-31T10:00:33Z","reason":"DesiredWithinRange","message":"the
        desired count is within the acceptable range"}]'
      autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":0,"currentAverageValue":"1m"}}]'
    creationTimestamp: "2019-12-31T09:49:16Z"
    name: php-apache
...
 spec:
    maxReplicas: 10
    minReplicas: 1
    scaleTargetRef:
      apiVersion: extensions/v1beta1
      kind: Deployment
      name: php-apache
    targetCPUUtilizationPercentage: 50
  status:
    currentCPUUtilizationPercentage: 0
    currentReplicas: 1
    desiredReplicas: 1

And checked again:

$ date && kubectl get hpa && kubectl get deployments php-apache 

Tue 31 Dec 2019 11:01:16 AM CET
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          12m

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
php-apache   1/1     1            1           97s

So now, I can see non-zero value for REPLICA and TARGETS shows correct value.

Hope that helps :) P.S. Let me know if you have solved the issue.