Git, sub-repos & external libs for web development - best strategy once and for all?

Question

This is such a common scenario that there must be a sensible solution, yet despite pages of reading and copious Git gymnastics, my brain aches and I'm unable to make this work...

I'm working with Wordpress, though this will fit most website dev scenario's. I want to manage the site install with a git repo and also manage various WP plugins, jQuery plugins and other code bits in separate repos which can easily be pulled/pushed from their external sources. Seems simple enough until you look at the details...

The criteria

"Subfolders" criterion The folder for each plugin should not be bound to the root folder of its source repo. Many repos have multiple nested folders such as "my-repo-name/...", "dev/", "test/", "src/" where the contents of the later is the thing required. This is important to keep the referencing URLs clean and to minimise publicly available rubbish.

"No Proxys" criterion The ideal solution would not require additional intermediary branches or repos. Pushing changes to a plugin's external source should be simple and not require multiple intermediary merges/pushes.

"Real Files" criterion Ideally the outer repo for the whole website should actually contain the files of the plugins' subrepos (ie no "submodules"). I could be persuaded away from this one though...

"Publishing" criterion It must play well with rsync and/or git push'ing to the live server

I've looked at these five solutions

Git Submodules Simple enough for making changes and pushing/pulling but submodules fail on the "Subfolders" and "Real File" criteria

Git read-tree/subtree merge Solves the "Real Files" problem and read-tree actually lets you reference a branch's subfolder but when I did so and tried to merge changes on master back upstream, Git failed to remember that it came from a subfolder and merged master's whole structure into the ext libs tracking branch...so FAIL on this criterion.

Apenwarrs subtree extension (here) Great for the "Real Files" criterion and fairly simple to push/pull until you want to enforce the "Subfolders" rule. At best it then seems to require intermediary branches where you split out the folder you want from the remote tracking branch and then add this as a subtree to your master branch. I didn't have much luck merging/pushing changes on master back to the source repo. I still think there might be possibility here...

Symbolic links with external repo Great solution until GIT stopped following symlinks. Now it fails on "Real Files" and "Publishing" criteria

Nested repos Somewhere I saw an SO answer where if you explicitly git add a folder which contains another repo and include the trailing slash, git will NOT submodule it but instead track the individual files. This seemed promising but it fails on the "Subfolders" criterion.

What next?

I've seen references to "sparse checkout" - or perhaps a something involving branch pruning. I'm hoping to avoid a solution that involves shell scripts or is so complex that it requires me to re-learn it every time (infrequent) I make a change to a plugin. It needs to be easier than maintaining a separate repo for each plugin and copying back and forth from the main CMS installation.

Surely someone has a simple functional way to make this common dev scenario work?? Thanks in advance for the help...

Answer 1

Clearly you've been thinking about this, and I am only a novice at git, but my first instinct would be to add .gitignore to the plugins directory and have each plugin have it's own git repo. You could do the same for themes. So I guess this is more of a question than an answer - why wouldn't this simple approach work?

Answer 2

I don't know if this will answer your question in full, or even cover your specific scenario, but I thought I'd give it a go in any case.

We have multiple projects, many of which reference each other, each in their own Git repository. They are also contained in multiple levels of folders, for example:

Repos
    - Utilities
        - Util repo1
        - Util repo2
    - Common
        - Lib repo1
        - Lib repo2
    - Projects
        - Client1
            - Git repo1
            - Git repo2
        - Client2 repo
        - Client3
        ...

We had an issue with manually having to push / pull / commit each repository, which became a nightmare.

So we wrote a little utility program that is able to do push, pull, commit, tag, remote diff and diff Git operations on multiple folders at a time, as well as build multiple VS solution files at the same time. It will also check in sub folders for repositories.

We have been using it for some time now, and it works fine for us. You can check it out from here: Gitter.7z

Open the ReadMe.html file to see how to use it. Hope this helps out, let me know.