How to blame previous versions of a specific line of code in VS 2015?

I'm using Visual Studio 2015 with Git integration. What I'm trying to achieve is going back in the history of a specific line of code.

I know there is Source Control > Annotate. But then I get only to see the latest change. Then, I can right-click the line on the annotations gutter and say "Annotate This Version" which will give me the latest change for that specific line. But from there, how to get back in history for that line? "Annotate This Version" is greyed out...

The "Annotate This Version" found in the context-menu of the left gutter works just as expected. In my case, the history didn't follow a file rename which was confusing me.