git svn clone large repo under Windows: out of memory - not a large file issue

Question

I am trying to clone a large svn repository with git svn. The repo has got 100000 revisions. The size is about 9GB (pristine folder). Biggest file in repo is 300 MB.

The branch structure is a total mess in the repo. Lots of wrong and missing merge info, no standard layout. I've tried to fetch the latest revisions with and without branches. The command without branches looks like this:

git svn clone url_to_trunk_in_repo -r100000:HEAD --username=svn_user

HEAD is currently at 101037. The process runs for a while (hours) and fails with something like this:

Out of memory during request for 29040 bytes, total sbrk() is 254959616 bytes!

I have got the latest maintained git revision for Windows (Git-1.9.4-preview20140929) running on Windows 7 x64 with 16 GB RAM.

I've done some search on this kind of failure. Most postings refer to a problem with large files some years ago which is most likely fixed already (haven't checked that). Anyway this issue refers to large allocation, indicated by the error message during "large" request. However, the process fails while adding normal implementation files of small size. Therefore, I don't think this is a large file problem.

I've tried to modify the pack settings in etc/gitconfig, which is a common advise. However, this didn't help. I didn't expect it to help at all because the memory error is during download from svn server not during git gc which processes the packs, AFAIK.

Further digging lead me to a perl memory limitation of 256MB. This is most likely the case because I always get the error with almost 256MB sbrk().

Further investigation on perl memory limitations brings up OS memory limitations, only. That is 2GB on win32 (3GB with special switch) and RAM limit for 64 bit windows. I also found some advice for raising Cygwin memory limitations but that doesn't apply here.

The 256MB limit is ridiculous in my eyes and I desperately searching for a way to get around this.

---

EDIT: This is propably a Perl 5.8.8 issue (git uses that version). I have also installed strawberry perl 5.16.3 x64. I've written this test code, which is a modification of the code posted at this stackoverflow question:

use strict;
use warnings;

my @s;
my $count = 200;
my $alloc = 30000000;

for (my $i = 0; $i < $count; $i++)
{   print "Trying allocation...";
    $s[$i] = "a" x $alloc;   # ok
    print "OK\n\n";
}

With strawberray perl, this works perfectly. In git bash, I receive the error described before.

Out of memory during "large" request for 33558528 bytes, total sbrk() is 2351800 32 bytes at mem.pl line 9.

---

EDIT 2: I've tried strawberry perl 5.8.8-1. It allocates properly, however, the program crashes after execution. Hence, this is not a bug in perl 5.8.8 generally but in the version that is being shipped with git (msys perl 5.8.8) Configuration of strawberry perl and msys perl differs in many entries. Most noticable difference for me is usemymalloc=n (strawberry) and usemymalloc=y (msys perl).

I also checked for ulimit in git bash, which doesn't show any abnormality:

$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 8
stack size              (kbytes, -s) 2046
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63
virtual memory          (kbytes, -v) 2097152

Answer 1

With Cygwin and Git 2.1.1 I'm able to run git svn on my repo without any memory issues. My test program runs fine as well. I haven't tried 1.x-Versions of Git on Cygwin but I guess they'd work because the problem was a memory limitation of msysperl which is replaced by Cygwin.

I won't mark this as answer since it doesn't solve my original question. It is my current workaround for tests with Git.

I'd like to have a Git for Windows distribution with properly working Perl. There is an issue for upgrading Perl here. However this seems to be not an easy task. Same holds for the SVN version used by git svn on Windows: Howto upgrade SVN