I have my Dockerfile, which looks like:
FROM confluentinc/cp-kafka-connect:4.0.0
ARG VERSION=0.0.2.15
RUN curl -Ls https://github.com/jcustenborder/kafka-connect-rabbitmq/releases/download/$VERSION/kafka-connect-rabbitmq-$VERSION.tar.gz | tar -xzC /tmp && \
mkdir -p /etc/kafka-connect/jars && \
cp -R /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/. /etc/kafka-connect/jars && \
ls -l /etc/kafka-connect/jars
RUN ls -l /etc/kafka-connect/jars
Now the issue I am facing is when I do that first ls
I can confirm that the JARs have been copied across as I'd expect.
However when I do that second ls
in the RUN
line, they are not shown (and subsequently they do not appear in my container when I run the image).
So I'd like to understand why is this happening? Am i wrong to assume after each line the image is passed along - as this doesnt seem the case.
FYI - this is the full output from my console:
Sending build context to Docker daemon 2.56kB
Step 1/4 : FROM confluentinc/cp-kafka-connect:4.0.0
---> 4db60f092134
Step 2/4 : ARG VERSION=0.0.2.15
---> Using cache
---> dc641b6beb04
Step 3/4 : RUN curl -Ls https://github.com/jcustenborder/kafka-connect-rabbitmq/releases/download/$VERSION/kafka-connect-rabbitmq-$VERSION.tar.gz | tar -xzC /tmp && cp /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/kafka-connect-rabbitmq-$VERSION.jar /usr/share/java/ && mkdir -p /etc/kafka-connect/jars && cp /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/*.jar /etc/kafka-connect/jars && ls -l /etc/kafka-connect/jars
---> Running in f48dbf0e487e
total 6804
-rw-r--r-- 1 root root 491199 Apr 2 03:48 amqp-client-4.2.0.jar
-rw-r--r-- 1 root root 74557 Apr 2 03:48 annotations-2.0.1.jar
-rw-r--r-- 1 root root 100811 Apr 2 03:48 connect-utils-0.3.101.jar
-rw-r--r-- 1 root root 7046 Apr 2 03:48 connect-utils-testing-data-0.3.101.jar
-rw-r--r-- 1 root root 1493680 Apr 2 03:48 freemarker-2.3.25-incubating.jar
-rw-r--r-- 1 root root 2256213 Apr 2 03:48 guava-18.0.jar
-rw-r--r-- 1 root root 55784 Apr 2 03:48 jackson-annotations-2.8.0.jar
-rw-r--r-- 1 root root 281079 Apr 2 03:48 jackson-core-2.8.5.jar
-rw-r--r-- 1 root root 1236315 Apr 2 03:48 jackson-databind-2.8.5.jar
-rw-r--r-- 1 root root 749499 Apr 2 03:48 javassist-3.19.0-GA.jar
-rw-r--r-- 1 root root 31212 Apr 2 03:48 kafka-connect-rabbitmq-0.0.2.15.jar
-rw-r--r-- 1 root root 129763 Apr 2 03:48 reflections-0.9.10.jar
-rw-r--r-- 1 root root 41071 Apr 2 03:48 slf4j-api-1.7.21.jar
Removing intermediate container f48dbf0e487e
---> ad2ca0767def
Step 4/4 : RUN ls -l /etc/kafka-connect/jars
---> Running in c0b5fda45249
total 0
Removing intermediate container c0b5fda45249
---> 5fef032d5aba
Successfully built 5fef032d5aba
Successfully tagged myfirstimage:latest
Do I have to use some sort of 'finalise' command?
Any help would be appreciated.
Thanks.
My guess is that the /etc/kafka-connect/jars
directory is declared as a VOLUME
in that image's Dockerfile.
And the output of the docker inspect
command confirms my guess:
$ docker image inspect confluentinc/cp-kafka-connect:4.0.0 --format '{{.Config.Volumes}}'
map[/etc/kafka-connect/secrets:{} /etc/kafka/secrets:{} /var/lib/kafka/data:{} /etc/kafka-connect/jars:{}]
Citing from The Dockerfile Specification:
If any build steps change the data within the volume after it has been declared, those changes will be discarded.
So, here are the details about your problem:
VOLUME /etc/kafka-connect/jars
.ls
command in this step works normally.The solution is to put the jar files on your host, and bind-mount the host directory to the container when running the container. As below:
docker run -v /path/contains/jar/files:/etc/kafka-connect/jars <IMAGE>
As @Yuankun mention is right but not only base image declare volume but also the current child image too. When inspect the child
docker image inspect test --format '{{.Config.Volumes}}'
I got the same output for as for base image
map[/etc/kafka-connect/jars:{} /etc/kafka-connect/secrets:{} /etc/kafka/secrets:{} /var/lib/kafka/data:{}]
So if you run the container and if you check ls in /etc/kafka-connect/jars
there will be nothing in that location that's why the second ls returns nothing.
The result is not due to removing the intermediate container but the behavior of cp in docker file during build time.
I made some changes in your docker file.
FROM confluentinc/cp-kafka-connect:4.0.0
ARG VERSION=0.0.2.15
RUN echo "Test again"
RUN curl -Ls https://github.com/jcustenborder/kafka-connect-rabbitmq/releases/download/$VERSION/kafka-connect-rabbitmq-$VERSION.tar.gz | tar -xzC /tmp
Run mkdir -p /etc/kafka-connect/jars && \
cp -R /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/. /etc/kafka-connect/jars && \
ls -l /etc/kafka-connect/jars
Run cp -R /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/.
/etc/kafka-connect/jars && \
ls -l /etc/kafka-connect/jars
WORKDIR /etc/kafka-connect/jars
ADD test.sh /usr/src/app/test.sh
RUN chmod +x /usr/src/app/test.sh
# ENTRYPOINT [ "/usr/src/app/test.sh" ]
When I build this docker image I got the following output. I run two-time cp and got correct output two times.
Step 1/9 : FROM confluentinc/cp-kafka-connect:4.0.0
---> 4db60f092134
Step 2/9 : ARG VERSION=0.0.2.15
---> Using cache
---> d341c91e6b25
Step 3/9 : RUN echo "Test again"
---> Running in 6fdd5aa8c36e
Test again
Removing intermediate container 6fdd5aa8c36e
---> 491238cfb53c
Step 4/9 : RUN curl -Ls https://github.com/jcustenborder/kafka-connect-rabbitmq/releases/download/$VERSION/kafka-connect-rabbitmq-$VERSION.tar.gz | tar -xzC /tmp
---> Running in 732784416c75
Removing intermediate container 732784416c75
---> 27efd005b0be
Step 5/9 : Run mkdir -p /etc/kafka-connect/jars && cp -R /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/. /etc/kafka-connect/jars && ls -l /etc/kafka-connect/jars
---> Running in ba93b4b53621
total 6804
-rw-r--r-- 1 root root 491199 Apr 2 06:10 amqp-client-4.2.0.jar
-rw-r--r-- 1 root root 74557 Apr 2 06:10 annotations-2.0.1.jar
-rw-r--r-- 1 root root 100811 Apr 2 06:10 connect-utils-0.3.101.jar
-rw-r--r-- 1 root root 7046 Apr 2 06:10 connect-utils-testing-data-0.3.101.jar
-rw-r--r-- 1 root root 1493680 Apr 2 06:10 freemarker-2.3.25-incubating.jar
-rw-r--r-- 1 root root 2256213 Apr 2 06:10 guava-18.0.jar
-rw-r--r-- 1 root root 55784 Apr 2 06:10 jackson-annotations-2.8.0.jar
-rw-r--r-- 1 root root 281079 Apr 2 06:10 jackson-core-2.8.5.jar
-rw-r--r-- 1 root root 1236315 Apr 2 06:10 jackson-databind-2.8.5.jar
-rw-r--r-- 1 root root 749499 Apr 2 06:10 javassist-3.19.0-GA.jar
-rw-r--r-- 1 root root 31212 Apr 2 06:10 kafka-connect-rabbitmq-0.0.2.15.jar
-rw-r--r-- 1 root root 129763 Apr 2 06:10 reflections-0.9.10.jar
-rw-r--r-- 1 root root 41071 Apr 2 06:10 slf4j-api-1.7.21.jar
Removing intermediate container ba93b4b53621
---> 530730db6dc3
Step 6/9 : Run cp -R /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/. /etc/kafka-connect/jars && ls -l /etc/kafka-connect/jars
---> Running in 6fdf56ea723a
total 6804
-rw-r--r-- 1 root root 491199 Apr 2 06:10 amqp-client-4.2.0.jar
-rw-r--r-- 1 root root 74557 Apr 2 06:10 annotations-2.0.1.jar
-rw-r--r-- 1 root root 100811 Apr 2 06:10 connect-utils-0.3.101.jar
-rw-r--r-- 1 root root 7046 Apr 2 06:10 connect-utils-testing-data-0.3.101.jar
-rw-r--r-- 1 root root 1493680 Apr 2 06:10 freemarker-2.3.25-incubating.jar
-rw-r--r-- 1 root root 2256213 Apr 2 06:10 guava-18.0.jar
-rw-r--r-- 1 root root 55784 Apr 2 06:10 jackson-annotations-2.8.0.jar
-rw-r--r-- 1 root root 281079 Apr 2 06:10 jackson-core-2.8.5.jar
-rw-r--r-- 1 root root 1236315 Apr 2 06:10 jackson-databind-2.8.5.jar
-rw-r--r-- 1 root root 749499 Apr 2 06:10 javassist-3.19.0-GA.jar
-rw-r--r-- 1 root root 31212 Apr 2 06:10 kafka-connect-rabbitmq-0.0.2.15.jar
-rw-r--r-- 1 root root 129763 Apr 2 06:10 reflections-0.9.10.jar
-rw-r--r-- 1 root root 41071 Apr 2 06:10 slf4j-api-1.7.21.jar
Removing intermediate container 6fdf56ea723a
---> ebe70157c1b9
Step 7/9 : WORKDIR /etc/kafka-connect/jars
Removing intermediate container 90bfc4ba1190
---> 6cc7949e289a
Step 8/9 : ADD test.sh /usr/src/app/test.sh
---> 0e98b550dac1
Step 9/9 : RUN chmod +x /usr/src/app/test.sh
---> Running in 07ceda6ab705
Removing intermediate container 07ceda6ab705
---> 37d4beaf3ba6
Successfully built 37d4beaf3ba6
Successfully tagged test:latest
Again when I run the container there is not a thing in that locations.
Solutions:
1st: Attach the volume that is created by docker build process.
docker run --rm -it -v /etc/kafka-connect/jars:/etc/kafka-connect/jars test bash
so here is the list of your jar files.
2nd: Copy these files at entry point script so it will persist in that location and you dont need to mount any local jars files and mount volume created by docker build.
create test.sh
cp -R /tmp/usr/share/kafka-connect/kafka-connect-rabbitmq/. /etc/kafka-connect/jars
exec "/bin/bash"
update your docker file.
FROM confluentinc/cp-kafka-connect:4.0.0
ARG VERSION=0.0.2.15
RUN curl -Ls https://github.com/jcustenborder/kafka-connect-rabbitmq/releases/download/$VERSION/kafka-connect-rabbitmq-$VERSION.tar.gz | tar -xzC /tmp
WORKDIR /etc/kafka-connect/jars
ADD test.sh /usr/src/app/test.sh
RUN chmod +x /usr/src/app/test.sh
ENTRYPOINT [ "/usr/src/app/test.sh" ]
Here we got what we need without doing any effort
docker run --rm -it test bash
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With