How to exec into a container and view file if container is in CrashLoopBackOff state

Question

I am running nodejs based application on kubernetes and it's in CrashLoopBackOff state.

kubectl logs api-5db677ff5-p824m

> [email protected] staging /home/node
> NODE_ENV=staging node src/loader.js

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] staging: `NODE_ENV=staging node src/loader.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] staging script.
npm ERR! This is probably not a problem with npm. There is likely additional 
logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2019-04-04T14_24_26_908Z-debug.log

There is not much information in logs and I cannot access the complete log file from the given path in container.

When I try check that file in container:

kubectl exec -it api-5db677ff5-p824m -- /bin/bash

It's giving me this error:

error: unable to upgrade connection: container not found ("api")

Answer 1

If you have access to the k8s node, you can access the logs of (all) the pods at /var/log/pods.

Alternatively, you can try to mount of PVC to your node pod and configure your node application to write to logs there. This will ensure that your crash logs are not destroyed when the container crashes.

Another similar approach is to override your pod container command with sleep 3600 and then exec into the container to run your node application manually. Once the node process crashes and writes the logs, you can then view them (inside the container).

Hope this helps!

Answer 2

You can call exec only for containers which are in a "running" state.

If a container in a CrashLoopBackOff state, you cannot do it, because you have no running container where K8s can call a command.

The only way to get that file is to save it to some shared volume or host directory and then check it there. Try to check Volumes documentation.