Cleaning git repository BFG Repo-Cleaner --- repo size is still large after I push

Question

I am trying to clean up my git repository with BFG repo cleaner. So I did: git clone --mirror https://github.com/FourierFlows/FourierFlows.jl.git used BFG, e.g., java -jar bfg-1.13.0.jar --delete-files 'twodturb*.png' FourierFlows.jl.git and then cd FourierFlows.jl.git git reflog expire --expire=now --all && git gc --prune=now --aggressive

The repository is now much smaller (1.9MB compared to 880MB). But when I pushed back and tried later to git clone --mirror again I still got the repository having the size I started up with.

navid:FourierFlows.jl.git/ (master) $ git push
Counting objects: 4824, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1639/1639), done.
Writing objects: 100% (4824/4824), 1.67 MiB | 1.07 MiB/s, done.
Total 4824 (delta 3061), reused 4824 (delta 3061)
remote: Resolving deltas: 100% (3061/3061), done.
To https://github.com/FourierFlows/FourierFlows.jl
 + f669d29...5146e5f ArchivedPhysics -> ArchivedPhysics (forced update)
 + 53da66f...6543491 BetteringTwoModes -> BetteringTwoModes (forced update)
 + 986cde0...a6562c4 BetteringTwoModesSystematically -> BetteringTwoModesSystematically (forced update)
 + 9b7e4e3...8db4ced BetteringTwoModes_StashedChanges -> BetteringTwoModes_StashedChanges (forced update)
 + 4adf9a4...bd7b62c GPUDomains -> GPUDomains (forced update)
 + 40e1318...bd7b4bf GPUTwoModes -> GPUTwoModes (forced update)
 + e2d551c...666d0f9 GPUTwoModes_APVTracerAdv -> GPUTwoModes_APVTracerAdv (forced update)
 + 23f1689...343ab15 TimestepperFormattingImprovements -> TimestepperFormattingImprovements (forced update)
 + 08bcc90...6a59cf1 TracerPatchEqn -> TracerPatchEqn (forced update)
 + 019c115...8b2b6c1 TwoModes -> TwoModes (forced update)
 + 740c9a1...c8c47dd UpdatesDocs -> UpdatesDocs (forced update)
 + 7456358...96c1686 VerticalModes -> VerticalModes (forced update)
 + 0b36327...a111d4e getETDcoeffsBigFLoat -> getETDcoeffsBigFLoat (forced update)
 + f90541a...de0c079 gh-pages -> gh-pages (forced update)
 + 8e1f439...f3e9ac0 makeJulia0p7compatible -> makeJulia0p7compatible (forced update)
 + 3211e48...8828ef5 master -> master (forced update)
 + 6ef5dfc...47c773c v0.0.1 -> v0.0.1 (forced update)
 + d6e2cf8...4e43e7f v0.0.2 -> v0.0.2 (forced update)
 + 3be73f7...dfc1060 v0.1.0 -> v0.1.0 (forced update)
 ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/11/head -> refs/pull/11/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/13/head -> refs/pull/13/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/14/head -> refs/pull/14/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/15/head -> refs/pull/15/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/head -> refs/pull/17/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/merge -> refs/pull/17/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/19/head -> refs/pull/19/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/22/head -> refs/pull/22/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/23/head -> refs/pull/23/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/23/merge -> refs/pull/23/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/24/head -> refs/pull/24/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/27/head -> refs/pull/27/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/30/head -> refs/pull/30/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/32/head -> refs/pull/32/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/33/head -> refs/pull/33/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/34/head -> refs/pull/34/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/35/head -> refs/pull/35/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/36/head -> refs/pull/36/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/40/head -> refs/pull/40/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/41/head -> refs/pull/41/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/42/head -> refs/pull/42/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/43/head -> refs/pull/43/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/45/head -> refs/pull/45/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/46/head -> refs/pull/46/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/47/head -> refs/pull/47/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/48/head -> refs/pull/48/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/48/merge -> refs/pull/48/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/49/head -> refs/pull/49/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/50/head -> refs/pull/50/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/52/head -> refs/pull/52/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/53/head -> refs/pull/53/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/54/head -> refs/pull/54/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/56/head -> refs/pull/56/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/57/head -> refs/pull/57/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/58/head -> refs/pull/58/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/9/head -> refs/pull/9/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/FourierFlows/FourierFlows.jl'

However, if I push the repository to an empty git repo and then clone it from there it is indeed small.

navid:FourierFlows.jl.git/ (master) $ git remote set-url --push origin https://github.com/navidcy/FourierFlowsMirrored
navid:FourierFlows.jl.git/ (master) $ git push
Counting objects: 4824, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1639/1639), done.
Writing objects: 100% (4824/4824), 1.67 MiB | 1.16 MiB/s, done.
Total 4824 (delta 3061), reused 4824 (delta 3061)
remote: Resolving deltas: 100% (3061/3061), done.
To https://github.com/navidcy/FourierFlowsMirrored
 * [new branch]      ArchivedPhysics -> ArchivedPhysics
 * [new branch]      BetteringTwoModes -> BetteringTwoModes
 * [new branch]      BetteringTwoModesSystematically -> BetteringTwoModesSystematically
 * [new branch]      BetteringTwoModes_StashedChanges -> BetteringTwoModes_StashedChanges
 * [new branch]      GPUDomains -> GPUDomains
 * [new branch]      GPUTwoModes -> GPUTwoModes
 * [new branch]      GPUTwoModes_APVTracerAdv -> GPUTwoModes_APVTracerAdv
 * [new branch]      TimestepperFormattingImprovements -> TimestepperFormattingImprovements
 * [new branch]      TracerPatchEqn -> TracerPatchEqn
 * [new branch]      TwoModes -> TwoModes
 * [new branch]      UpdatesDocs -> UpdatesDocs
 * [new branch]      VerticalModes -> VerticalModes
 * [new branch]      getETDcoeffsBigFLoat -> getETDcoeffsBigFLoat
 * [new branch]      gh-pages -> gh-pages
 * [new branch]      makeJulia0p7compatible -> makeJulia0p7compatible
 * [new branch]      master -> master
 * [new tag]         v0.0.1 -> v0.0.1
 * [new tag]         v0.0.2 -> v0.0.2
 * [new tag]         v0.1.0 -> v0.1.0
 ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/11/head -> refs/pull/11/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/13/head -> refs/pull/13/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/14/head -> refs/pull/14/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/15/head -> refs/pull/15/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/head -> refs/pull/17/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/merge -> refs/pull/17/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/19/head -> refs/pull/19/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/22/head -> refs/pull/22/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/23/head -> refs/pull/23/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/23/merge -> refs/pull/23/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/24/head -> refs/pull/24/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/27/head -> refs/pull/27/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/30/head -> refs/pull/30/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/32/head -> refs/pull/32/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/33/head -> refs/pull/33/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/34/head -> refs/pull/34/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/35/head -> refs/pull/35/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/36/head -> refs/pull/36/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/40/head -> refs/pull/40/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/41/head -> refs/pull/41/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/42/head -> refs/pull/42/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/43/head -> refs/pull/43/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/45/head -> refs/pull/45/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/46/head -> refs/pull/46/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/47/head -> refs/pull/47/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/48/head -> refs/pull/48/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/48/merge -> refs/pull/48/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/49/head -> refs/pull/49/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/50/head -> refs/pull/50/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/52/head -> refs/pull/52/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/53/head -> refs/pull/53/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/54/head -> refs/pull/54/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/56/head -> refs/pull/56/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/57/head -> refs/pull/57/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/58/head -> refs/pull/58/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/9/head -> refs/pull/9/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/navidcy/FourierFlowsMirrored'

Any ideas why is that? Apparently somehow I'm not telling git to ditch the files that are up in the repository already...

Answer 1

This is a common problem with cleaning GitHub repos. The key to the problem is:

! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)

So all history in Pull Requests which has been cleaned is not being cleaned in the GitHub remote.

This is discussed briefly in the GitHub documentation on how to remove sensitive data (emphasis mine):

This article tells you how to make commits with sensitive data unreachable from any branches or tags in your GitHub repository. However, it's important to note that those commits may still be accessible in any clones or forks of your repository, directly via their SHA-1 hashes in cached views on GitHub, and through any pull requests that reference them. You can't do anything about existing clones or forks of your repository, but you can permanently remove all of your repository's cached views and pull requests on GitHub by contacting GitHub Support.

So if you want your Pull Requests, and all the commits/history they reference wiped, so your total repo size on GitHub can shrink (in addition to the size of default clones, which have already shrunk), you need to contact GitHub Support.