Why does Git not automatically fast forward?

Question

When I do git pull, add, commit, push, this happens

 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'http://foo:8080/tfsdev/foo/_git/Development.Services'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

when I add -ff to git pull, it goes through. I thought fastforward was default, why would this happen?

Answer 1

When I add -ff to git pull, it goes through. I thought fastforward was default, why would this happen?

You are mixing up a bunch of different concepts here ... which is no surprise, as git pull also mixed up a bunch of different concepts, in an attempt to be convenient that mostly results in being confusing. (Well, also, occasionally convenient. :-) )

Using git pull does usually run git merge, which often does do a fast-forward instead of a merge. This brings your branch up to date with whatever git pull brought over from the remote (via git fetch) so that commits you add, will also only add to (not replace or remove-from) their commits. But git push is not the opposite of git pull, it's the opposite of git fetch: it does not do any merging.

When you do a pull + do stuff (commit or whatever) + push sequence, you are racing against other people who are also doing pull-commit-push sequences. Whoever wins, gets to push. This part really is that simple. The problem comes in when you don't win!

It's easy to win this race when you're the only one running. If no one else is pushing to that branch, you will always win the race. What happened here is that you were not the only one running, and you lost.

The full answer is, unfortunately, complicated, and you did not show enough of what was going on to be able to answer this one specific case. So let's take a look at what really happens.

Always keep track of your current branch

If git status says on branch master, or git branch prints * master, your current branch is master. Most Git commands work with your current branch, and git pull is no exception here.

The git pull command simply runs two other Git commands. The first one is always git fetch. The second one, you can change.

First, git pull runs git fetch

The first command git pull runs is git fetch. It runs a somewhat limited git fetch, telling git fetch to bring over only as many commits and other items as are needed for your current branch. If you just run git fetch, Git will bring over everything, which potentially takes a little bit longer, but means you're up to date with, well, everything.

The git fetch command is an exception to the usual Git rule about working with your current branch. It does still look at your current branch in some cases, but what git fetch does normally is to update your remote-tracking branches. In most cases you will have exactly one so-called remote, named origin, and running git fetch brings over everything from origin and updates your origin/master, origin/develop, and other origin/ branches. These are your remote-tracking branches for the remote named origin.

Hence, if you run git fetch yourself, it will update all your origin/* remote-tracking branches. If you run git pull, it will run git fetch in a way that updates only one of these, based on your current branch. These updates are always safe: they bring new commits into your repository, which have no effect on all the existing commits in your repository.

Just to be confusing, git fetch writes all its updates to a file named FETCH_HEAD, which you will see mentioned in the git fetch documentation. But it also updates your remote-tracking branches, unless you have a very old version of Git (older than Git version 1.8.4). For those with ancient Gits (1.7.x), a manual git fetch of everything from origin updates them, but the one run by git pull doesn't.

Then git pull runs something else

Here things get a bit messy. The second part of git pull is either git merge or git rebase. The default is git merge, even though git rebase is often a better idea. You must choose in advance which command git pull should use. But which one should you use? The only way to tell for sure is to look at what is coming in. If you pick up the new commits, you can look at them with git log and other Git commands.

To see what's coming in, though, you have to bring it in. That's what git fetch does. The git fetch step brings in the new commits, safely, putting them safely behind those remote-tracking branches. Now you can decide whether to merge or to rebase.

The git pull command makes you decide before you look. So pick which one you want it to do—merge, or rebase—and pull, and it will fetch and then do that. To make it do a rebase, run git pull --rebase, or set the current branch to make git pull do that by default.

(This setting is per branch, so you must set it for each branch. There's also a configuration entry telling Git to set "rebase mode" automatically for each branch Git creates—but it's a lot easier, in my opinion, to just run git fetch yourself, and then run the second command yourself. This gives you the chance to look at what you've fetched, as well as being a lot less confusing when things go wrong.)

When the second command is git merge

Let's assume that the second thing git pull runs is git merge. This does work on your current branch. But git merge is itself somewhat complicated.

When you merge, you pick some commit—usually by a branch name, and very often by remote-tracking branch names like origin/master—and ask Git to figure out several things. The goal of this merge is easy to state: A merge combines work you did with work other people did, to make a new commit with both sets of work.

To figure out what you did, and what they (whoever they are) did, Git has to find the merge base. This merge base is, more or less, the first commit that you and they share. This means it's time to draw part of a commit graph. Here are some possibilities.

The fast-forward "merge"

...--o--*     <-- master
         \
          o   <-- origin/master

The os represent commits, with newer commits towards the right. I have marked the merge base commit with *. Here, your master points to the merge base commit, with origin/master pointing to a newer commit.

In this case, git merge is able to do this "fast forward" thing. That is, because the merge base already is the tip of master, there's no actual merging required. Git can just slide the name master down-and-forward so that it points to the same commit as origin/master:

...--o--o
         \
          o   <-- master, origin/master

Now we can straighten out the line as well, and just have:

...--o--o--o  <-- master, origin/master

If you now run git push origin master, basically nothing happens, because there's nothing new on your end. That last commit is one you already got from origin.

The real merge

But what if we have this instead?

...--o--*--o   <-- master
         \
          `-o    <-- origin/master

Here, Git has to do a real merge. Git compares commit * to your master commit (the rightmost o on the top line), and to your origin/master commit (the o on the bottom line). That is, Git runs git diff twice. Then it tries to combine the two sets of changes and, if it succeeds, it makes a new commit, of the "merge commit" flavor:

...--o--*--o--o   <-- master
         \   /
          `-o    <-- origin/master

This commit is ready to git push to origin, in a race against anyone else also trying to git push to origin.

Or maybe there's nothing at all to do

You could even have this to start with:

...--o--o--*   <-- master, origin/master

Now there's nothing for you to merge: the name master and the name origin/master both point to the same commit, so the merge base commit is already the tip of master (and of origin/master as well). Now there's nothing to merge, and after doing nothing, there's nothing to push either. You don't even have to attempt the race, much less win it.

Now you can make new commits too

Now that your master and origin/master line up, you can make new commits. When you do, your master will "get ahead of" origin/master:

...--o--o   o--o   <-- master
         \ /
          o   <-- origin/master

(I moved the bottom o down just so that I could point origin/master to it.)

What git push does

I mentioned above that push is not the opposite of pull; it's more the opposite of fetch.

When you run git push, your Git calls up some other Git. That's the Git on the remote origin, in this case. It then hands over any new commits you've made, and a request: "please set your master to this latest commit."

In other words, we ask their Git to change their master–the thing we call origin/master—to match our master.

Look at those same drawings above. When we did a fast-forward, we moved our master to a commit that we had already picked up from origin/master. When we did nothing, we did nothing. In either case, there is no new commit: there's nothing to bother pushing. If we do push anyway, we'll say to the other Git: "There's nothing new here, but please set your master so that it points to this particular commit, just like ours does."

If we did make a new commit—as we did with the merge that really did something, or when we made new commits—we do have new stuff for them. We'll hand them our new commits, then ask them to set their master the same as our master.

Of course, we'll be racing anyone else also pushing.

If no one else has won the push race, their master will still match our origin/master. We'll ask to push, and—well, just look at that drawing above. Moving origin/master (our name for their master) forward to match our master, why, that's one of those same fast-forward operations we saw earlier!

If a push will do a fast-forward, it will generally succeed.

But what if someone else won? What if, after we ran git fetch and git merge, we spent some time making commits, and meanwhile someone else got in and pushed before us?

Well, now we have this:

...--o--o   o--o   <-- master
         \ /
          o   <-- origin/master

but that's not really quite right, because they have new commits on their master. They have the commits someone else pushed. We need to run git fetch and get them. Let's do that, and now—now that git fetch updated our origin/master with the new commits—now we have:

...--o--o   o--o   <-- master
         \ /
          o-----o   <-- origin/master

Now, in order to push, we must either merge or rebase. I'll draw the merge case. (Note that the merge base is the left-most commit on the bottom line.)

...--o--o   o--o--o   <-- master
         \ /     /
          *-----o   <-- origin/master

If we make this merge, now we can push, because now our master is "fast-forward ahead of" their master.

Of course, while we're merging, we are also racing other people who might be trying to push. If we lose the race, we may need to fetch and merge again.

(In all of these situations, git rebase is probably a better thing to do than git merge. See other StackOverflow answers about rebase.)