unity3d and git submodules is it possible?

Question

TLDR; This is going to be a long winded post but I am sure many of you unity3d developers are also running into the same problem I am. A problem that needs a clear definitive, once and for all answer to save our collective sanity.

So I have been using git for the last 2+ years but I have not dived deep into it too much. I can branch/merge push pull from bitbucket/github etc and that works fine for regular win forms/traditional applications.

Here is the problem. I've long since moved from xna/silverlight over to unity3d and in unity in order to add a reference to a library you actually have to copy the source files from that library into your unity project folder. Although unity does allow you to drop *.dll files into a Plugin folder, there are obvious cross platform compatibility issues, and thus I never want to go there.

I have over a dozen library projects providing various functionality from AI, to content management, to logging, abstracted user input system etc etc. and I have been working on a new level editor for a future game I will be making.

Each individual library project it self is a unity project and the code files for each project are organized in unity as such

Assets\<company name>\<project name>\
Assets\<company name>\<library name>\
Assets\<company name>\<another library name>\

I do this to keep things highly organized unlike many of the other assets that are on the unity asset store. :(

So my level editor project is using multiple library projects whose code has been imported via copy paste. Each of those libraries are themselves hosted in there own local git repositories. So when I copy paste the code files from the library projects to my main unity project I automatically loose any ability to commit changes I make to that libraries git repo.

What I have been doing this last year has been the fallowing

1. Start a main project Assets\<company name>\<main project name>\
2. Import any library projects (copy/paste code files) Assets\<company name>\<library name>\
3. If I need to make a change to one of the libraries I usually just edit the libraries code file(s) that are in the main project that I am working on. (otherwise more complexity ensues)
4. Because the main project is also a git repo the changes I make to any library files also gets committed to the main project's repo.
5. After a while I use CodeCompare (a diff utility) to do a folder comparison of Assets\<company name>\<library name>\ with the original location of the library where I previously copied the library files from.
6. I painstakingly compare each changed code file diff by diff and migrate changes made in my main project over to the original location of the library.
7. Now that that is done, the library is a git repo so again (2nd time) I have to make commits about the various changes that were made to the code files.

Along with that insanity here lies yet another problem. The main project is not the only main project I have on the go right now. I have a handful of projects that all reference (via copy/paste) these other libraries. So now that I have updated the code for the library and committed it I now have to go back through each main project that uses the library and copy/paste over the changes from the libraries git repo over to those main projects.

Work flow summerized

MainProjectA->Code files copied from LibraryA->LibraryA files modified inside MainProjectA->LibraryA files merged via diff comparison back to original LibraryA folder location->LibraryA code files then copied back to other projects that use LibraryA IE: MainProjectB, MainProjectC, MainProjectD.

Good lord! As my various projects get larger and larger this problem gets compounded more and more. This system works but it is very very tedious.

Intermission

So a bit of a short rant. /TakesDeepBreath /sigh I refuse to use the git command line. It wreaks of MS-DOS ala 1980's. It's 20 freaking 14 people. 2014 and us developers are still geeking out with command lines. In this day and age, I expect more. I am not learning basically a whole new command line language let alone have to switch back and fourth between windows and apps just to do something as fundamentally simple like commit a code change. Something that can be accomplished with 2-3 fast mouse clicks and a short typed message is all that is needed. git = new hawtness + unnecessary command line complexity

Having said that I most often use the built in vs2013 git features for making commits and branching/merging etc. But it does not appear to support submodules, as far as I can tell. So for that I use SourceTree.

End Intermission.

Enter git submodule hell

Like I said I have not yet used git a lot beyond it's basic core functionality and I have only done preliminary tests via test projects but I can't get git to work reliably the way I need it to and often when working with submodules there is a bit of confusion, and it often times will not commit because of various errors.

What I want to do is this

1. Create new unity project called MainProjectA and make it a local repo
2. Add git submodule(s) to MainProjectA from LibraryA (local repo), LibraryB (local repo) etc
3. If I make changes to code files related to LibraryA inside of MainProjectA I want the ability to commit those specific changes back to LibraryA's repo from within MainProjectA.

I have even gone so far as to begin writing a utility that would automatically sync files between 2 different folders as soon as it detects a change. But I abandoned it because although it would solve synchronization issues it would not solve the git commit issues.

Another problem is that all my existing unity projects that have a local repo are setup as fallows

<Unity Main Project Folder>
  |- .git
  |- Assets
    |-Company Name
      |- MainProjectA
  |- Library
  |- ProjectSettings
  |- .gitignore

<Unity LibraryA Folder>
  |- .git
  |- Assets
    |-Company Name
      |- LibraryA
  |- Library
  |- ProjectSettings
  |- .gitignore

What I want to do is have a LibraryA folder in my main project that is a submodule of my LibraryA project, more specifically I want only the contents of <Unity LibraryA Folder>\Assets\Company Name\LibraryA\ to be included as a submodule in my main project folder <Unity Main Project Folder>\Assets\Company Name\LibraryA\

<Unity Main Project Folder>
  |- .git
  |- Assets
    |-Company Name
      |- MainProjectA
      |- LibraryA (submodule)
  |- Library
  |- ProjectSettings
  |- .gitignore

But SourceTree/git complains that I can't add a submodule to a existing repo folder. <Unity Main Project Folder>\Assets\Company Name\

I was going to continue writing but ... yeah I think that's enough writing, and my goal should be described well enough by now.

Conclusion

I guess I could also ask weather git has matured enough as a product that it will even allow me to do what I am trying to do? Or is it simply not possible given the architecture of git to support what I am trying to do?

Every search has come up empty, YouTube is no help and a waste of time, not to mention every example I can find is a command line example, and does not touch on submodules.

In a normal windows application submodules would not be a problem. The problem is specifically with how unity likes to have all the code files in it's project. That and how I am trying to keep my organized folder structure intact.

Please help. Please. Purdy please. I BEG YOU! I BEG YOU! ON MY FRIEAKIN KNEES BEGGING! :P

Answer 1

There is a really good explanation and process on the Prime31 website : http://prime31.github.io/A-Method-for-Working-with-Shared-Code-with-Unity-and-Git/ This is in my opinion the best workflow to handle this situation.

Answer 2

Maybe you want to use symlinks/hardlinks?

<Master Project Folder>   <----- no .git here
|
|-<Unity Main Project Folder>
|   |- .git
|   |- Assets
|     |-Company Name
|       |- MainProjectA
|       |- LibraryA         ------ symlink to --+
|       |- .gitignore   <--------+              |
|   |- Library                   |              |
|   |- ProjectSettings          add LibraryA    |
|   |- .gitignore                               |
|                                               |
|-<Unity LibraryA Folder>                       |
|   |- .git                                     |
|   |- Assets                                   |
|     |-Company Name                            |
|   |- LibraryA       <-------------------------+
|   |- Library
|   |- ProjectSettings
|   |- .gitignore

This should work, despite Unity complaining about symlinks. At least on OSX.

---

Another (and IMO more correct) way is to have LibraryA as a submodule and have a separate test project for it (currently your Unity LibraryA Folder).

I have done 2+ years in this setup (2-3 linked repositories) and git has performed well. Maybe your graphical client is not so good? Try gitx-dev or github's desktop client (which works for non-github repos too) - or just learn the command line, it works.

Oh and the problems with adding a submodule to an existing directory - remove the directory first and commit.