I want to add a 9.7 GB project to GitHUB

Question

I am trying to push my local git into my remote but the project is too big:

MINGW64 ~/source/repos/DBO (master)
$ git config http.postBuffer 9370000000

MINGW64 ~/source/repos/DBO (master)
$ git push --set-upstream origin master
Enumerating objects: 149849, done.
Counting objects: 100% (149849/149849), done.
Delta compression using up to 16 threads
Compressing objects: 100% (94939/94939), done.
fatal: protocol error: bad line length 81925 MiB | 342.52 MiB/s
send-pack: unexpected disconnect while reading sideband packet
error: failed to push some refs to 'https://github.com/Alejandro-HUB/DBO.git'

MINGW64 ~/source/repos/DBO (master)

I tried using this to increase GitHUB's file size limit but I think they may have a limit for how big the project can be:

$ git config http.postBuffer 9370000000

Is there any way for me to get my project into GitHUB?

Answer 1

You can push such a project to GitHub, but only incrementally. GitHub has a 2 GB limit on all pushes, which cannot be changed. You can push it incrementally by doing something like this to push 5000 commits at a time:

$ git rev-list --reverse HEAD | \
  perl -ne 'print unless $i++ % 5000;' | \
  xargs -I{} git push origin {}:master

Once that completes, most of the objects will be pushed, and you should be able to push the rest of the data.

If you have a single commit which introduces data that's more than 2 GB, then you won't be able to push that to GitHub. You'll have to split it up into smaller commits.

Also, the Git FAQ explains why you shouldn't need to set http.postBuffer and why doing so will waste a huge amount of memory. You should unset that value.