docker add cache when git checkout same file

Question

I need checkout project in my CI server build image reusing docker cache.

Docker ADD not use cache when checkout same file.

I am in git branch A execute docker build -t somename . it use docker cache normally, but if I go to branch B via git checkout B, do nothing, go to branch A via git checkout A and run again docker build -t somename . the docker cache is using only before the first ADD.

here example:

Dockerfile

# DOCKER-VERSION 0.10.0
FROM myregistry:5000/ruby:2.1.2-1
MAINTAINER ME

# Gem sources
RUN gem source -r https://rubygems.org/
RUN gem source -a http://gems.mydomain

# Never install a ruby gem docs
RUN echo "gem: --no-rdoc --no-ri" >> ~/.gemrc

# gems install
RUN mkdir /foo
WORKDIR /foo

RUN gem install bundler
ADD Gemfile /foo/Gemfile
RUN bundle install

# expose ports
EXPOSE 8080

log from build

Sending build context to Docker daemon 19.54 MB
Sending build context to Docker daemon 
Step 0 : FROM myregistry:5000/ruby:2.1.2-1
 ---> 9ce683a713b4
Step 1 : MAINTAINER ME
 ---> Using cache
 ---> 8f54114fd2e7
Step 2 : RUN gem source -r https://rubygems.org/
 ---> Using cache
 ---> f58a08708863
Step 3 : RUN gem source -a http://gems.mydomain
 ---> Using cache
 ---> 3e69e17c5954
Step 4 : RUN echo "gem: --no-rdoc --no-ri" >> ~/.gemrc
 ---> Using cache
 ---> 1edb37962dd4
Step 5 : RUN mkdir /foo
 ---> Running in 3d3441c34ee3
 ---> aeb90f00bc9b
Removing intermediate container 3d3441c34ee3
Step 6 : WORKDIR /foo
 ---> Running in 84b881d8f621
 ---> 10e1d8984458
Removing intermediate container 84b881d8f621
Step 7 : RUN gem install bundler
 ---> Running in 31e98523ce46
Successfully installed bundler-1.6.2
1 gem installed
 ---> 84d5195ab831
Removing intermediate container 31e98523ce46
Step 8 : ADD Gemfile /foo/Gemfile
 ---> 3e5f2675ee22
Removing intermediate container c90e8be5ea17
Step 9 : RUN bundle install
 ---> Running in ac0e83e5eebb    
Fetching gem metadata from http://gems.mydomain/......
Fetching additional metadata from http://gems.mydomain/..
Resolving dependencies...
Installing rake 10.3.2
Installing i18n 0.6.9
Installing multi_json 1.10.1
.
.
.
Installing railties 3.2.19
Installing responders 0.9.3
Using bundler 1.6.2
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.    
 ---> 19beae703adb
Removing intermediate container ac0e83e5eebb
Step 10 : EXPOSE 8080
 ---> Running in 1b1e55d349e5
 ---> 32405bdac6d1
Removing intermediate container 1b1e55d349e5
Successfully built 32405bdac6d1

git checkout B

git checkout A

docker build -t somename .

second build log

Sending build context to Docker daemon 19.52 MB
Sending build context to Docker daemon 
Step 0 : FROM myregistry:5000/ruby:2.1.2-1
 ---> 9ce683a713b4
Step 1 : MAINTAINER ME
 ---> Using cache
 ---> 8f54114fd2e7
Step 2 : RUN gem source -r https://rubygems.org/
 ---> Using cache
 ---> f58a08708863
Step 3 : RUN gem source -a http://gems.mydomain
 ---> Using cache
 ---> 3e69e17c5954
Step 4 : RUN echo "gem: --no-rdoc --no-ri" >> ~/.gemrc
 ---> Using cache
 ---> 1edb37962dd4
Step 5 : RUN mkdir /foo
 ---> Using cache
 ---> aeb90f00bc9b
Step 6 : WORKDIR /foo
 ---> Using cache
 ---> 10e1d8984458
Step 7 : RUN gem install bundler
 ---> Using cache
 ---> 84d5195ab831
Step 8 : ADD Gemfile /foo/Gemfile
 ---> 4977e35c80f7
Removing intermediate container bd59cc0d5e51
Step 9 : RUN bundle install
 ---> Running in 6ff16f32e94a
Fetching gem metadata from http://gems.mydomain/......
Fetching additional metadata from http://gems.mydomain/..
Resolving dependencies...
Installing rake 10.3.2
Installing i18n 0.6.9
Installing multi_json 1.10.1
.
.
.    
Installing railties 3.2.19
Installing responders 0.9.3
Using bundler 1.6.2
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
 ---> d9332f9035c3
Removing intermediate container 6ff16f32e94a
Step 10 : EXPOSE 8080
 ---> Running in b20252a00160
 ---> 4d9932882e06
Removing intermediate container b20252a00160
Successfully built 4d9932882e06

Answer 1

Docker invalidates the docker build cache when a file's mtime value has changed, and git does not track file mtime values. This cache invalidation also shows up in other situations too, like in continuous integration or build environments involving docker, git and branches.

I have been using a "touch" target in a Makefile that I run prior to asking docker to build a container:

touch:
    @echo "Reset timestamps on git working directory files..."
    find ./ | grep -v .git | xargs touch -t 200001010000.00

Next, always run make touch prior to docker build or any docker-based "build" target in the same Makefile...

Another option is to setup a git hook that modifies mtime values automatically: https://git.wiki.kernel.org/index.php/ExampleScripts#Setting_the_timestamps_of_the_files_to_the_commit_timestamp_of_the_commit_which_last_touched_them

Another possible solution is to fork docker and remove mtime from its definition of caches: https://github.com/docker/docker/blob/master/pkg/tarsum/tarsum.go

Note: as of docker 1.8, mtime is no longer taken into account when invalidating the cache. Pull request #12031 updated this behavior