How to delete stale pods which are in Pending state for more than n hours?

Question

I am using the following command to gracefully delete any stale pods in Pending state:

kubectl get pod -n my-namespace | grep Pending | awk '{print $1}' | xargs kubectl delete pod -n my-namespace

However, I would like to add a condition that deletes only those pods who have been in pending state for more than N hours. There is the AGE column returned with get pods but its time unit varies and I am assuming there is a better way.

Also would appreciate if anyone can mention any best practices around this as I aim to run this command periodically to cleanup the pending Pods.

Answer 1

It is difficult to calculate how much time a Pod has spent in a particular Status by using kubectl solely and without a help of some 3rd party tools. However, I got a solution that you may find useful.

You can list all Pods that are in Pending state and are older than X days. For example, the command below will list all Pending Pods that are older than 5 days:

kubectl get pods --field-selector=status.phase=Pending --sort-by=.metadata.creationTimestamp | awk 'match($5,/[6-9]d|[0-9][0-9]d|[0-9][0-9][0-9]d/) {print $0}'

Than you can use the next command to delete these pods:

kubectl delete pod $(kubectl get pods --field-selector=status.phase=Pending --sort-by=.metadata.creationTimestamp | awk 'match($5,/[6-9]d|[0-9][0-9]d|[0-9][0-9][0-9]d/) {print $0}')

The value can and should be adjusted by modifying the awk scripting in order to match your use case.