Rapidly modifying Python app in K8S pod for debugging

Question

---

Background

I have a large Python service that runs on a desktop PC, and I need to have it run as part of a K8S deployment. I expect that I will have to make several small changes to make the service run in a deployment/pod before it will work.

---

Problem

So far, if I encounter an issue in the Python code, it takes a while to update the code, and get it deployed for another round of testing. For example, I have to:

• Modify my Python code.
• Rebuild the Docker container (which includes my Python service).
• scp the Docker container over to the Docker Registry server.
• docker load the image, update tags, and push it to the Registry back-end DB.
• Manually kill off currently-running pods so the deployment restarts all pods with the new Docker image.

This involves a lot of lead time each time I need to debug a minor issue. Ideally, I've prefer being able to just modify the copy of my Python code already running on a pod, but I can't kill it (since the Python service is the default app that is launched, with PID=1), and K8S doesn't support restarting a pod (to my knowledge). Alternately, if I kill/start another pod, it won't have my local changes from the pod I was previously working on (which is by design, of course; but doesn't help with my debug efforts).

---

Question

Is there a better/faster way to rapidly deploy (experimental/debug) changes to the container I'm testing, without having to spend several minutes recreating container images, re-deploying/tagging/pushing them, etc? If I could find and mount (read-write) the Docker image, that might help, as I could edit the data within it directly (i.e. new Python changes), and just kill pods so the deployment re-creates them.

---

Answer 1

There are two main options: one is to use a tool that reduces or automates that flow, the other is to develop locally with something like Minikube.

For the first, there are a million and a half tools but Skaffold is probably the most common one.

For the second, you do something like ( eval $(minikube docker-env) && docker build -t myimagename . ) which will build the image directly in the Minikube docker environment so you skip steps 3 and 4 in your list entirely. You can combine this with a tool which detects the image change and either restarts your pods or updates the deployment (which restarts the pods).

Also FWIW using scp and docker load is very not standard, generally that would be combined into docker push.