Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Managing large binary files with Git

I am looking for opinions of how to handle large binary files on which my source code (web application) is dependent. We are currently discussing several alternatives:

  1. Copy the binary files by hand.
    • Pro: Not sure.
    • Contra: I am strongly against this, as it increases the likelihood of errors when setting up a new site/migrating the old one. Builds up another hurdle to take.
  2. Manage them all with Git.
    • Pro: Removes the possibility to 'forget' to copy a important file
    • Contra: Bloats the repository and decreases flexibility to manage the code-base and checkouts, clones, etc. will take quite a while.
  3. Separate repositories.
    • Pro: Checking out/cloning the source code is fast as ever, and the images are properly archived in their own repository.
    • Contra: Removes the simpleness of having the one and only Git repository on the project. It surely introduces some other things I haven't thought about.

What are your experiences/thoughts regarding this?

Also: Does anybody have experience with multiple Git repositories and managing them in one project?

The files are images for a program which generates PDFs with those files in it. The files will not change very often (as in years), but they are very relevant to a program. The program will not work without the files.

like image 764
pi. Avatar asked Feb 12 '09 08:02

pi.


People also ask

How does Git handle large files?

When you have source files with large differences between versions and frequent updates, you can use Git LFS to manage these file types. Git LFS is an extension to Git which commits data describing the large files in a commit to your repo, and stores the binary file contents into separate remote storage.

Does Git compress binary files?

It can, literally, compress (or "deltify") any binary data against any other binary data—but the results will be poor unless the inputs are well-chosen. It's the input choices that are the real key here. Git also has a technical documentation file describing how objects are chosen for deltification.

Is Git LFS good?

LFS is More Complexity Version control should just work. Large file handling should just work. End-users shouldn't have to care that large files are handled slightly differently from small files. The usability of Git LFS is generally pretty good.


1 Answers

I discovered git-annex recently which I find awesome. It was designed for managing large files efficiently. I use it for my photo/music (etc.) collections. The development of git-annex is very active. The content of the files can be removed from the Git repository, only the tree hierarchy is tracked by Git (through symlinks). However, to get the content of the file, a second step is necessary after pulling/pushing, e.g.:

$ git annex add mybigfile $ git commit -m'add mybigfile' $ git push myremote $ git annex copy --to myremote mybigfile ## This command copies the actual content to myremote $ git annex drop mybigfile ## Remove content from local repo ... $ git annex get mybigfile ## Retrieve the content ## or to specify the remote from which to get: $ git annex copy --from myremote mybigfile 

There are many commands available, and there is a great documentation on the website. A package is available on Debian.

like image 132
rafak Avatar answered Sep 27 '22 16:09

rafak