Maintaining git repo containing other cloned git repo

Question

I am working on a website using Sharelatex(github) but it contains other repositories that are used to build the main project. I cloned the main repository and did grunt install which is used to download those repositories.

But the problem is that I need to change the code in both main repository and the downloaded ones.

Since these projects can get new updates I want to merge those changes as well. I also need to maintain a repo, but when I push changes to Github it only shows changes in the main repo.

I came across submodules in git but since the main project does not contain any submodule types I am unable to use that.

For example:

There is a repo web used in the main project. I begin with some edits in the files in web. I need these changes to reflect in my remote repository so that others can use them.

Now suppose after some time an important update for web repo is available, how should I use that?

I enter grunt install in the command line to download this repo. It does not create a submodule but clones that repo into my folder, which is subsequently ignored by my main git repo.

The question might seem unclear but I tried my best to explain the problem.

In the nut-shell:

• I want not only to perform changes in both main and any other repo that's involved, but pull and merge those other repos, as well, when their updates are available.

• I also need to maintain a remote repo of my project.

Answer 1

I am working on a website using Sharelatex(github) but it contains other repository that are used to build the main project. [...] I came across submodules in git but since the main project does not contain any submodule kind of thing I am unable to use that.

Actually, you do have a submodule. That's what a nested repository is, and (I'm serious about this) so far as what you need to understand to grok submodules, that's all there is to submodules. To understand submodules, imagine you have a nested repository (you do) and think about the administrative requirements, what needs to be done to support that setup in a dvcs.

For starters, when people clone a project that uses submodules from some repo you've all decided contains published commits, that clone obviously won't also get the subproject repository (certainly it shouldn't get your private and god-knows-what-you've-done-to-it version). So they also have to get the subproject repo from some one of its own published repositories.

How do you tell people fetching your commits where to get the necessary subproject commits? Plainly, you have to drop a note somewhere in a committed file saying "here's a repo that should have any needed $subproject commits". git submodule has settled on .gitmodules as the conventional place to store notes like this.

Next up: well, what are others to do if the url you handed them goes offline? They're plainly going to need to use another repo. Hence, .gitmodules is just suggestions, the git submodule command uses the current values in your .git/config, which git submodule init has populated from the suggested ones in .gitmodules`.

git submodule's operations are all like this. Forget about it. Don't even bother looking at the command until you need a little assistance doing what you have already found needs doing. Start from the knowledge, the simple fact, that a submodule is nothing more than a nested repo, and the project using it commits nothing more than a commit id that's supposed to be somewhere in that nested repo. That's it. That's all a submodule is.

As you run across tedious tasks that need doing, look for a git submodule subcommand that does them for you. You don't have to use the subcommand. All that subcommand is doing is automating straightforward tasks that would be laborious otherwise. It's a toolkit for doing whatever it is you need to do, and there's no way on earth it could or should impose some arbitrary and sufficient (<-- that's the hard part) abstraction on everybody in the world. So it's a grab-bag.

That said, there is one important safety play git submodule update and git submodule add perform for you when they do the git clone for you. Repositories conventionally have the actual repo content under the [sub]project toplevel .git, but if you check out a branch that doesn't have that subproject there or otherwise need or want that subproject gone, its .git will also go away -- not what you want when it holds not just your checked out content but the entire actual repo. So when git submodule update does its initial clone, it hoists the submodule's .git directory into a handy (and arbitrary) little nook in the containing project's repo, and replaces the .git directory it just moved out of the submodule with a .git file containing the relative path to the moved directory.

To get that initial hoisting done on the repo you currently have, move it out of your current repo, add and update from wherever you put it, and then fix up the upstream urls in .gitmodules for the convenience of others.

There. Now you know absolutely everything you need to know to understand git submodules and incrementally acquire details only as you find you need them, to understand what the git submodule command is doing for you, and why you don't actually have to care about understanding every little thing on its manpage up front. At least, I think so.

If I've missed anything important, I'd be very glad of (gentle or blunt, I really don't care) corrections in the comments.