Git pull fatal: Out of memory, malloc failed

Question

I've a repo on https://bitbucket.org/

Few days ago by a mistake big number of image files were pushed in the repo. then files were deleted via another push. after that repo worked ok, but today when i try to pull from the repo:

$ git pull
Password for 'https://[email protected]': 
warning: no common commits
remote: Counting objects: 4635, done.
remote: Compressing objects: 100% (1710/1710), done.
fatal: Out of memory, malloc failed (tried to allocate 4266852665 bytes)
fatal: index-pack failed  

I've tried:
1) git config --global pack.windowMemory 1024m
2)

$ git count-objects -v
count: 9
size: 48
in-pack: 4504
packs: 1
size-pack: 106822
prune-packable: 0
garbage: 0

No luck there, not sure what actions should i take next...
The size of the repo should be around 10-20m of code. what actions should i take next?

UPDATE 1
i executed these commands:

$ git filter-branch --index-filter 'git rm --cached --ignore-unmatch public/images/*' HEAD
Rewrite a1c9fb8324a2d261aa745fc176ce2846d7a2bfd7 (288/288)
WARNING: Ref 'refs/heads/master' is unchanged

and

$ git push --force --all
Counting objects: 4513, done.
Compressing objects: 100% (1614/1614), done.
Writing objects: 100% (4513/4513), 104.20 MiB | 451 KiB/s, done.
Total 4513 (delta 2678), reused 4500 (delta 2671)
remote: bb/acl: ayermolenko is allowed. accepted payload.
To https://[email protected]/repo.git
 + 203e824...ed003ce demo -> demo (forced update)
 + d59fd1b...a1c9fb8 master -> master (forced update)

Pull then works ok:

$ git pull
Already up-to-date.

But when i try to clone repo i get

~/www/clone$ git clone [email protected]:repo.git
Cloning into 'clone'...
remote: Counting objects: 5319, done.
remote: Compressing objects: 100% (1971/1971), done.
fatal: Out of memory, malloc failed (tried to allocate 4266852665 bytes)
fatal: index-pack failed

UPDATE 2
Sadly enough i didn't find all of the large files. some are still left. So i asked support to kill all the logs of the repo

UPDATE 3
In the end i had to kill old & create new repo.

Answer 1

In my case it was something as simple as trying to pull a big repo in a 1GB RAM box without swap.

I followed this tutorial https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04 to create some swap space on the server and worked.

Their "faster" way:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

You can make these changes permanent by adding to /etc/fstab:

/swapfile   none    swap    sw    0   0

They recommend adding to /etc/sysctl.conf:

vm.swappiness=10
vm.vfs_cache_pressure = 50

Answer 2

If you are the only one using this repo, you can follow the git filter-branch option described in "How to purge a huge file from commits history in Git?"

The simpler option is cloning the repo to an old commit, and force push it, as described in "git-filter-branch to delete large file".

Either one would force any collaborator to reset his/her own local repo to the new state you are publishing. Again, if you are the only collaborator, it isn't an issue.