How to copy directory out of Kubernetes job right when the job completes?

Question

The kubectl cp command only seems to work when the pod is still running. Is there a way to copy a directory of output files from a completed pod to my local machine?

Answer 1

By definition not a completed Pod, as those are ephemeral, however the answer to your question is to change the definition of what "completed" means.

The most straightforward answer to your question is to either mount a network Volume into the Pod, so its files survive termination, or to have the Pod copy its own files out to some extra-cluster location (maybe s3, or an FTP site).

But I suspect you don't mean under those those circumstances, or you would have already done so.

One other example might be to have the Pod wait some defined timeout period for the appearance of a sentinel file so you can inform to the Pod you have successfully copied the files out and that it is now free to terminate as expected. Or if it's more convenient, have the Pod listen on a socket and stream a tar (or zip) to a connection, enabling a more traditional request-response lifecycle, and at the end of the response then the Pod shuts down.

Implied in all those work-around steps is that you are notified of the "almost done"-ness of the Pod through another means. Without more information about your setup, it's hard to go into that, and might not even be necessary. So feel free to add clarifications as necessary.