Remove a lot of files in git filter-branch

Question

I'm migrating a repository from svn to git.

In this last step, I want to remove tons of files that aren't needed from the history.

I'm trying the following command:

git filter-branch --prune-empty --index-filter \
  "for file in $(cat files); do git rm -rf --cached --ignore-unmatch ${file}; done" -f

But it says that the argument list is too long.

I could rewrite this like:

for file in $(cat files); do
  git filter-branch --prune-empty --index-filter \
    "git rm -rf --cached --ignore-unmatch ${file}" -f
done

But it will run filter-branch tons of times, and the history is long.. so, it would take too much time.

Is there a faster way to filter-branch removing lots of files?

Answer 1

I'd recommend using The BFG, a simpler, faster alternative to git-filter-branch specifically designed for removing unwanted files from Git history.

You mentioned in your comment that the problem files are generally big binaries, and The BFG has a specific option for handling this - you should carefully follow the BFG's usage instructions, but the core part is just this:

$ java -jar bfg.jar  --strip-blobs-bigger-than 10M  my-repo.git

Any files over 10MB in size (that aren't in your latest commit) will be removed from your Git repository's history. You can then use git gc to clean away the dead data:

$ git gc --prune=now --aggressive

The BFG is typically at least 10-720x faster than running git-filter-branch, and generally easier to use.

Full disclosure: I'm the author of the BFG Repo-Cleaner.