Incrementally saving Clojure Data structures in a Git Repo

Question

Note:

This is tagged both Clojure and Git on purpose, as I'm happy to re-engineer on both the Clojure and the Git side of the software to make things work.

Problem:

I have a bunch of clojure data structures. They average around 100 kb each when written out to disk (with pr). I have about 1000 of such files. These files are "structured" documents -- think of them as equivalent to SVG.

Now, I am making lots of small updates to each of these files (adding/deleting nodes, changing property on nodes). (then I (write (pr ...)) these files out.

Lastly, I'm storing all of these files in my git repo.

Question:

I'm wondering if there is an efficient way to store these files (since different writes only have small modifications) -- i.e. if I have two copies of a single document in memory, it'd be 1MB + epsilon, not 2MB (since the two docs have only slight differences, and share most of the structure.)

I would like to somehow exploit this fact, and when it is being stored into git, have this similarity be taken advantage of.

Possible considered solutions:

1) On the clojure side, instead of writing out an entire file, write only a "diff [consisting of assoc, dissoc] from the previous file." <-- This requires lots of engineering.

2) On the FS side, instead of storing individual files, throw the entire directory into bzip then commit it as a single *.bz2 (thus, similar files will ahve similar blocks). Down side is that stroing *.bz2 files in git does not seem like a bad idea.

Answer 1

Have you considered using Datomic rather than Git?

What you are doing sounds like a roughly ideal use case for Datomic, which is essentially a Clojure-style database of "fact graphs" - which is pretty much exactly what your structured data look likes.

Datomic stores data in a manner very similar to Clojure's immutable data structures, i.e. it uses structural sharing to ensure that small changes only require a small amount of additional space. And it keeps the entire history in a manner similar to git - there's even a tool that simulates git repos in Datomic (codeq)