Menu
There aren't any other pods deployed. I have 2 nodes with zero pods. Its a t3.small instance type.
Is there any other commands to debug and see why it is still stuck in pending state?
kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-6cc489dcd5-5jc8t 0/1 Pending 0 91s
kubectl describe pod mysql-6cc489dcd5-5jc8t
Name: mysql-6cc489dcd5-5jc8t
Namespace: default
Priority: 0
Node: <none>
Labels: app=mysql
pod-template-hash=6cc489dcd5
Annotations: kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/mysql-6cc489dcd5
Containers:
mysql:
Image: mysql:5.6
Port: 3306/TCP
Host Port: 0/TCP
Environment:
MYSQL_ROOT_PASSWORD: dbpassword11
Mounts:
/docker-entrypoint-initdb.d from usermanagement-dbcreation-script (rw)
/var/lib/mysql from mysql-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nlx4p (ro)
Volumes:
mysql-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ebs-mysql-pv-claim
ReadOnly: false
usermanagement-dbcreation-script:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: usermanagement-dbcreation-script
Optional: false
default-token-nlx4p:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-nlx4p
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Deployments shows that it has scaled and nothing wrong there.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 5m34s deployment-controller Scaled up replica set mysql-6cc489dcd5 to 1
cat 01-storage-class.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
cat 02-persistent-volume-claim.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-mysql-pv-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: ebs-sc # Metadata name of the storage class
resources:
requests:
storage: 4Gi
cat 03-UserManagement-ConfigMap.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: usermanagement-dbcreation-script
data:
mysql_usermgt.sql: |-
DROP DATABASE IF EXISTS usermgmt;
CREATE DATABASE usermgt;
cat 04-mysql-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
name: mysql-deployment
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:5.6
env:
- name: MYSQL_ROOT_PASSWORD
value: dbpassword11
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage # name from the volumes listed below
mountPath: /var/lib/mysql
- name: usermanagement-dbcreation-script # name from the volumes listed below
mountPath: /docker-entrypoint-initdb.d
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: ebs-mysql-pv-claim
- name: usermanagement-dbcreation-script
configMap:
name: usermanagement-dbcreation-script
cat 05-mysql-clusterip-service.yml
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
#type: # Defaults to cluster IP service
selector:
app: mysql
ports:
- name: mysql
port: 3306 # If name isn't provided, port can start as a list `- port: 3306`
#targetPort # Defaults to containerPort
clusterIP: None # Use POD IP directly
kubectl describe nodes
Name: ip-192-168-14-238.ap-south-1.compute.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=eksdemo1
alpha.eksctl.io/nodegroup-name=eksdemo1-ng-public1
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t2.small
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=eksdemo1-ng-public1
eks.amazonaws.com/nodegroup-image=ami-01cfab2ec59e55c25
eks.amazonaws.com/sourceLaunchTemplateId=lt-02527e7be817e66a5
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=ap-south-1
failure-domain.beta.kubernetes.io/zone=ap-south-1a
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-192-168-14-238.ap-south-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t2.small
topology.ebs.csi.aws.com/zone=ap-south-1a
topology.kubernetes.io/region=ap-south-1
topology.kubernetes.io/zone=ap-south-1a
Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0b3699bcac4e70d31"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 25 Nov 2020 05:05:47 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: ip-192-168-14-238.ap-south-1.compute.internal
AcquireTime: <unset>
RenewTime: Thu, 26 Nov 2020 01:41:17 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:06:17 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.14.238
ExternalIP: 15.206.94.127
Hostname: ip-192-168-14-238.ap-south-1.compute.internal
InternalDNS: ip-192-168-14-238.ap-south-1.compute.internal
ExternalDNS: ec2-15-206-94-127.ap-south-1.compute.amazonaws.com
Capacity:
attachable-volumes-aws-ebs: 39
cpu: 1
ephemeral-storage: 20959212Ki
hugepages-2Mi: 0
memory: 2039136Ki
pods: 11
Allocatable:
attachable-volumes-aws-ebs: 39
cpu: 940m
ephemeral-storage: 18242267924
hugepages-2Mi: 0
memory: 1551712Ki
pods: 11
System Info:
Machine ID: 10f1040dde9f47289c7e10ea3fb9f040
System UUID: EC2C067A-2E0E-8A49-76C1-7F5FFA16E7ED
Boot ID: d2638886-d5b3-4fa6-a15b-e95e046397f5
Kernel Version: 4.14.203-156.332.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.6
Kubelet Version: v1.18.9-eks-d1db3c
Kube-Proxy Version: v1.18.9-eks-d1db3c
ProviderID: aws:///ap-south-1a/i-0b3699bcac4e70d31
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-zn9v5 10m (1%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system coredns-6b4cdc67b4-pzrl7 100m (10%) 0 (0%) 70Mi (4%) 170Mi (11%) 20h
kube-system ebs-csi-controller-7bf8974dc8-cth9n 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system ebs-csi-node-2vr6j 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system kube-proxy-dd4wd 100m (10%) 0 (0%) 0 (0%) 0 (0%) 20h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 210m (22%) 0 (0%)
memory 70Mi (4%) 170Mi (11%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
Name: ip-192-168-57-231.ap-south-1.compute.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=eksdemo1
alpha.eksctl.io/nodegroup-name=eksdemo1-ng-public1
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t2.small
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=eksdemo1-ng-public1
eks.amazonaws.com/nodegroup-image=ami-01cfab2ec59e55c25
eks.amazonaws.com/sourceLaunchTemplateId=lt-02527e7be817e66a5
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=ap-south-1
failure-domain.beta.kubernetes.io/zone=ap-south-1b
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-192-168-57-231.ap-south-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t2.small
topology.ebs.csi.aws.com/zone=ap-south-1b
topology.kubernetes.io/region=ap-south-1
topology.kubernetes.io/zone=ap-south-1b
Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0317d2c406ad6034d"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 25 Nov 2020 05:05:49 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: ip-192-168-57-231.ap-south-1.compute.internal
AcquireTime: <unset>
RenewTime: Thu, 26 Nov 2020 01:41:17 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:06:20 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.57.231
ExternalIP: 52.66.246.122
Hostname: ip-192-168-57-231.ap-south-1.compute.internal
InternalDNS: ip-192-168-57-231.ap-south-1.compute.internal
ExternalDNS: ec2-52-66-246-122.ap-south-1.compute.amazonaws.com
Capacity:
attachable-volumes-aws-ebs: 39
cpu: 1
ephemeral-storage: 20959212Ki
hugepages-2Mi: 0
memory: 2039136Ki
pods: 11
Allocatable:
attachable-volumes-aws-ebs: 39
cpu: 940m
ephemeral-storage: 18242267924
hugepages-2Mi: 0
memory: 1551712Ki
pods: 11
System Info:
Machine ID: a14b97ab355f485294e0e6bd736775af
System UUID: EC2D6C5A-497D-E6A4-F235-CF3055FCB546
Boot ID: c37c79e9-4bb1-4ffb-810f-80a4405cb403
Kernel Version: 4.14.203-156.332.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.6
Kubelet Version: v1.18.9-eks-d1db3c
Kube-Proxy Version: v1.18.9-eks-d1db3c
ProviderID: aws:///ap-south-1b/i-0317d2c406ad6034d
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-qnjzl 10m (1%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system coredns-6b4cdc67b4-whszf 100m (10%) 0 (0%) 70Mi (4%) 170Mi (11%) 20h
kube-system ebs-csi-controller-7bf8974dc8-cs6p8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system ebs-csi-node-2qwjq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system kube-proxy-pkxc9 100m (10%) 0 (0%) 0 (0%) 0 (0%) 20h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 210m (22%) 0 (0%)
memory 70Mi (4%) 170Mi (11%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
766
asked Nov 26 '20 06:11
If a Pod is Running but not Ready it means that the Readiness probe is failing. When the Readiness probe is failing, the Pod isn't attached to the Service, and no traffic is forwarded to that instance.
In this setup, I would first suspect something wrong with the persistent storage. Do a describe on the Persistent Volume Claim and Storage Class to see whether they are alright.
Also, if this is an EKS cluster, you can use the preinstalled EBS storage class.
54
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
