I am trying to setup HPA for my AKS cluster. Following is the Kubernetes manifest file:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: XXXXXX\tools\kompose.exe
convert
kompose.version: 1.21.0 (992df58d8)
creationTimestamp: null
labels:
io.kompose.service: loginservicedapr
name: loginservicedapr
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: loginservicedapr
strategy: {}
template:
metadata:
annotations:
kompose.cmd: XXXXXX\kompose.exe
convert
kompose.version: 1.21.0 (992df58d8)
creationTimestamp: null
labels:
io.kompose.service: loginservicedapr
spec:
containers:
image: XXXXXXX.azurecr.io/loginservicedapr:latest
imagePullPolicy: ""
name: loginservicedapr
resources:
requests:
cpu: 250m
limits:
cpu: 500m
ports:
- containerPort: 80
resources: {}
restartPolicy: Always
serviceAccountName: ""
volumes: null
status: {}
---
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: XXXXXXXXXX\kompose.exe
convert
kompose.version: 1.21.0 (992df58d8)
creationTimestamp: null
labels:
io.kompose.service: loginservicedapr
name: loginservicedapr
spec:
type: LoadBalancer
ports:
- name: "5016"
port: 5016
targetPort: 80
selector:
io.kompose.service: loginservicedapr
status:
loadBalancer: {}
Following is my HPA yaml file:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: loginservicedapr-hpa
spec:
maxReplicas: 10 # define max replica count
minReplicas: 3 # define min replica count
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: loginservicedapr
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Pods
pods:
name: cpu
target:
type: Utilization
averageUtilization: 50
But when HPA is failing with the error 'FailedGetResourceMetric' - 'missing request for CPU'.
I have also installed metrics-server (though not sure whether that was required or not) using the following statement:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
But still I am getting the following output when I do 'kubectl describe hpa':
Name: loginservicedapr-hpa
Namespace: default
Labels: fluxcd.io/sync-gc-mark=sha256.Y6dHhIOs-hNYbDmJ25Ijw1YsJ_8f0PH3Vlruj5rfbFk
Annotations: fluxcd.io/sync-checksum: d5c0d9eda6db0c40f1e5e23e1356d0268dbccc8f
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{"fluxcd.io/sync-checksum":"d5c0d9eda6db0c40f1e5...
CreationTimestamp: Wed, 08 Jul 2020 17:19:47 +0530
Reference: Deployment/loginservicedapr
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 50%
Min replicas: 3
Max replicas: 10
Deployment pods: 3 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 33m (x1234 over 6h3m) horizontal-pod-autoscaler Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: missing request for cpu
Warning FailedGetResourceMetric 3m11s (x1340 over 6h3m) horizontal-pod-autoscaler missing request for cpu
I have 2 more services that I have deployed along with 'loginservicedapr'. But I have not written HPA for those services. But I have included resource limits for those services as well in their YAML files. How to make this HPA work?
Horizontal pod autoscaler Effectively, the HPA is updated every 60 seconds. When changes are required, the number of replicas is increased or decreased accordingly. Horizontal pod autoscaler works with AKS clusters that have deployed the Metrics Server for Kubernetes 1.8+.
The Horizontal Pod Autoscaler changes the shape of your Kubernetes workload by automatically increasing or decreasing the number of Pods in response to the workload's CPU or memory consumption, or in response to custom metrics reported from within Kubernetes or external metrics from sources outside of your cluster.
According to the Kubernetes HPA Docs , HPA ‘calculates the utilization value as a percentage of the equivalent resource request on the containers in each pod’. In addition, the HPA Autoscaling Algorithm states that ‘CPU utilization is the recent CPU usage of a pod (average across the last 1 minute) divided by the CPU requested by the pod’.
If the pod is running, it can also be useful to check the logs of the pods and the containers that are inside them. Run the following series of kubectl logs commands: Is the pod running? In this case, test the connectivity by starting a test pod in the cluster.
This error indicates a subnet in use for a cluster no longer has available IPs within its CIDR for successful resource assignment. For Kubenet clusters, the requirement is sufficient IP space for each node in the cluster. For Azure CNI clusters, the requirement is sufficient IP space for each node and pod in the cluster.
You either need to install the heapster (Deprecated) or the metrics-server minimally to be able to use an HPA. This provides the minimum set of CPU and Memory metrics to be able to autoscale. A good way to see if you have either installed is that if you get this kind output from kubectl top pod:
resources
appears twice in your pod spec.
resources: # once here
requests:
cpu: 250m
limits:
cpu: 500m
ports:
- containerPort: 80
resources: {} # another here, clearing it
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With