Menu
All the pods of a node are on Evicted state due to "The node was low on resource: ephemeral-storage."
portal-59978bff4d-2qkgf 0/1 Evicted 0 14m
release-mgmt-74995bc7dd-nzlgq 0/1 Evicted 0 8m20s
service-orchestration-79f8dc7dc-kx6g4 0/1 Evicted 0 7m31s
test-mgmt-7f977567d6-zl7cc 0/1 Evicted 0 8m17s
anyone knows the quick fix of it.
819
Please consider following factors: Application that you are deploying via Kubernetes should have limits and requests set for memory and CPU in manifest file. As per your application requirements you should have your nodes configured in Kubernetes cluster. Increase no of nodes if all of them are heavily used by apps.
Ephemeral storage is the volatile temporary storage attached to your instances which is only present during the running lifetime of the instance. In the case that the instance is stopped or terminated or underlying hardware faces an issue, any data stored on ephemeral storage would be lost.
You can use /bin/df as a tool to monitor ephemeral storage usage on the volume where ephemeral container data is located, which is /var/lib/kubelet and /var/lib/containers .
Kubernetes uses ephemeral volumes to handle the storage needs of these transient pods. Ephemeral volumes allow pods to be started and stopped without being limited to the location of persistent volume. Kubernetes nodes have local ephemeral storage which is attached to the RAM or local writable devices.
Pods that use emptyDir volumes without storage quotas will fill up this storage, where the following error is present:
eviction manager: attempting to reclaim ephemeral-storage
Set a quota limits.ephemeral-storage, requests.ephemeral-storage to limit this, as otherwise any container can write any amount of storage to its node filesystem.
A sample resource quota definition
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources
spec:
hard:
pods: "4"
requests.cpu: "1"
requests.memory: 1Gi
requests.ephemeral-storage: 2Gi
limits.cpu: "2"
limits.memory: 2Gi
limits.ephemeral-storage: 4Gi
Another reason for this issue can be log files eating disk space. Check this question
80
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
