Git BFG to retroactively enable LFS - protected commits issue

Question

I have large files and was attempting to use the new Git LFS system.

I posted this question - Git lfs - "this exceeds GitHub's file size limit of 100.00 MB"

Edward Thomson correctly identified my issue - you cannot use LFS retroactively. He suggested I use BFG LFS support

This worked to a degree. The vast majority of my files were changed. However, there were protected commits that were not altered.

Of these protected commits some were over 100.00MB and thus caused a remote:error from github

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit c7cd871b (protected by 'HEAD') - contains 165 dirty files :
    - Directions_api/Applications/LTDS/Cycling/Leisure/l__cyc.csv (147.3 KB)
    - Directions_api/Applications/LTDS/Cycling/Work/w_cyc.csv (434.0 KB)
    - ...

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.

First of all - can someone explain why these commits are protected and different from those that BFG successfully changed?

Secondly - how can I unprotect these and allow BFG to edit them, thus allowing me to use LFS correctly and finally push successfully to GitHub

Answer 1

The BFG was originally created as a tool to destroy unwanted data (big files, passwords) buried deep in Git history. Data would be gone after the BFG had run, and programs often need to be adapted to handle this data no longer being directly available (ie they have to be changed to read passwords from environment variables, or to download big dependencies). Adapting programs to handle these changes is not a step that can be easily automated - humans need to update the code to cope with that change!

I took a decision to assume by default that projects were 'reformed' - they had made mistakes in their past, but by the time the user had come to discover the BFG they had already realised their mistakes and would at least have cleaned up their latest commits (ie they would already have made a commit deleting the unwanted data, as a first step to tackling the problem). Thus it was safer for the BFG not to change the latest commit - the alternative was for the BFG to blindly stomp everything in history, including the latest version of the project, and actually break software that wasn't yet ready to handle reading it's passwords from env variables, etc.

This behaviour can be disabled by adding a --no-blob-protection flag, eg:

$ java -jar ~/bfg-1.12.7.jar --convert-to-git-lfs '*.{exe,dll}' --no-blob-protection

I plan to update the BFG to implicitly use the --no-blob-protection flag when --convert-to-git-lfs is the only operation being performed - because this is no longer a truly destructive operation.

Full disclosure: I'm the author of the BFG Repo-Cleaner.