Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

kubectl cp fails with "tar: this does not look like a tar archive" on nodes running nvidia-docker

I'm trying to copy a local Python file to a running container on Kubernetes and it fails:

$ kubectl cp /path/to/file.py namespace/pod:/path/in/container/file.py
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
command terminated with exit code 2

I know the tar binary must be available in the container, and it is.

Does anyone know what's going on here and how I can solve this?

UPDATE:

After some more testing I can confirm that this only happens on nodes that run nvidia-docker rather than the normal docker. When piping things into kubectl exec on these nodes the stream is always empty.

So the following command yields an empty file in the pod running on a GPU-enabled node while the file is non-empty on other nodes without GPU support:

cat nonempty_file.txt | kubectl exec -i pod -- tee /home/jovyan/empty_file.txt 

This has been tested using the exact same image/container on both nodes.

like image 822
Simon G. Avatar asked Oct 21 '19 04:10

Simon G.


People also ask

What is kubectl cp command?

The kubectl cp command is used to copy files between containers and the user machine or cloud instance. To copy files from a container, Kubernetes first creates a tar file inside the container, copies it over the network, and then kubectl unpacks it on the user's machine or cloud instance.

How do I transfer files to POD using kubectl?

# Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the default namespace kubectl cp /tmp/foo_dir <some-pod>:/tmp/bar_dir # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container kubectl cp /tmp/foo <some-pod>:/tmp/bar -c <specific-container> # Copy /tmp/foo local file to /tmp/ ...

Does kubectl cp overwrite?

it will only override the destination different_file. txt which is already present in the destination /tmp/test directory.

Which command is used to receive output from a command run on the first container in a pod?

kubectl will connect to your cluster, run /bin/sh inside the first container within the demo-pod pod, and forward your terminal's input and output streams to the container's process.


1 Answers

Problem is solved by updating EKS AMI version.

Please install new release: eks-ami-releases.

See: eks-ami-kubectl.

like image 55
Malgorzata Avatar answered Oct 10 '22 04:10

Malgorzata