Version control system for huge files?

Question

I am looking for a quick but not-so-dirty way to do snapshots of a bunch of files totaling about 80 gigs. The issue here is that many of the files are around 1 GB large.

What is the best free version control system for this type of thing?

I know ZFS is an option, but I'd rather try something else first.

Answer 1

Subversion will handle your > 1GB files with good natured aplomb for the most part but if there are many large changes expect the generation of diffs to take a while...

Subversion Best practices has a section on large files:

A nice feature of Subversion is that by design, there is no limit to the size of files it can handle. Files are sent "streamily" in both directions between Subversion client and server, using a small, constant amount of memory on each side of the network.

Of course, there are a number of practical issues to consider. While there's no need to worry about files in the kilobyte-sized range (e.g. typical source-code files), committing larger files can take a tremendous amount of both time and space (e.g. files that are dozens or hundreds of megabytes large.)

To begin with, remember that your Subversion working copy stores pristine copies of all version-controlled files in the .svn/text-base/ area. This means that your working copy takes up at least twice as much disk space as the original dataset. Beyond that, the Subversion client follows a (currently unadjustable) algorithm for committing files:

. Copies the file to .svn/tmp/ (can take a while, and temporarily uses extra disk space))

. Performs a binary diff between the tmpfile and the pristine copy, or between the tmpfile and an empty-file if newly added. (can take a very long time to compute, even though only a small amount of data might ultimately be sent over the network)

. Sends the diff to the server, then moves the tmpfile into .svn/text-base/

So while there's no theoretical limit to the size of your files, you'll need to be aware that very large files may require quite a bit of patient waiting while your client chugs away. You can rest assured, however, that unlike CVS, your large files won't incapacitate the server or affect other users.

Answer 2

You might really want to try Monotone though, just check it out. You might find what you are looking for with it.