How do I use oauth2 and refresh tokens with the google api?

Question

So I just spent the last few days trying to figure this out and am asking this question so that I can answer it for other people who are having problems.

First, the google documentation is TERRIBLE and uses different oauth2 libraries depending on which of the many google API examples you are looking at. It is often self-contradictory and sometimes straight up has code in it that doesn't work.

Oh well.

so my questions were basically:

1. how do I use the google api libraries to have my users grant me access to their google accounts?
2. how do I store the oauth2 access tokens that google returns so that I can use them a few days from now?
3. how do i actually use the refresh_token and refresh it?

see answer below for a fully functional authorization flow, from getting the initial token to saving it, loading it later, refreshing it, and using it.

Cheers.

Answer 1

first, the google docs on how to use their API are terrible and self-contradictory.

heres my solution (using their libraries) to using oauth2 to using tokens that I store in a database and refresh periodically. I'm using django 2.0 and python 3.6. this is all in my 'views.py' file.

first, the imports and other script-wide setttings:

import google.oauth2.credentials
import google.auth.transport.requests
import google_auth_oauthlib.flow
from googleapiclient.discovery import build

import os
import json
import datetime

API_SCOPE = ['https://mail.google.com/',]
JSON_FILE = "test_server_client_json.json"
JSON_PATH = os.path.join(os.getcwd(),"<folder_name>",JSON_FILE)
if settings.TEST_SERVER:
    REDIRECT_URI = "http://localhost:5000/oauth2/gmail/"
    #we don't have ssl on local host
    os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
else:
    REDIRECT_URL = "https://www.example.com/oauth2/gmail/"

Ok, and here is the first server endpoint/page that we send the user to start the auth process

@login_required
def connect_gmail_to_manager_page_1(request):
    #this is the function that a new user uses to set up their gmail account
    #and connect it to our system.
    #this particular page is used to:
    #1) have the user enter their email address so we know what is going on
    #2) explain the process
    #=====================    
    #basically we get their email address, and thats it, on this page. then     we send them 
    #to google to grant us access.
    if request.method == "POST":
        form = admin.getEmailAddress(request.POST)
        if form.is_valid():
            #first, get their email address. this is optional.
            #i'm using django and their forms to get it.
            new_email = form.cleaned_data.get("email")
            #-----
            #we are going to create the flow object using <redacted>'s keys and such
            flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file(
                JSON_PATH,
                scopes=API_SCOPE)

            flow.redirect_uri = REDIRECT_URI

            # Generate URL for request to Google's OAuth 2.0 server.
            # Use kwargs to set optional request parameters.
            authorization_url, state = flow.authorization_url(
                # Enable offline access so that you can refresh an access     token without
                # re-prompting the user for permission. Recommended for web     server apps.
                access_type='offline',
                #which email is trying to login?
                login_hint=new_email,
                # Enable incremental authorization. Recommended as a         best     practice.
                include_granted_scopes='true')

            #and finally, we send them off to google for them to provide     access:
            return HttpResponseRedirect(authorization_url)
    else:    
        form = admin.getEmailAddress()
    token = {}
    token.update(csrf(request))
    token['form'] = form
    return render(request,'connect_gmail_to_manager_page_1.html',token)

This sends the user off to google to give us authorization. after they grant it, the user is redirected to an authorization endpoint on our servers. Here is my authorization endpoint (i've removed some project-specific code here)

@login_required
def g_auth_endpoint(request):
    #this is the endpoint that the logged in token is sent to
    #here we are basically exchanging the auth code provided by gmail for an     access token.
    #the access token allows us to send emails.
    #it is a passthrough endpoint: we want to redirect to the next stage of 
    #whatever process they are doing here on completion.
    #===============================================
    #first we need to get the paramater 'state' from the url
    #NOTE that you should do some error handling here incase its not a valid token. I've removed that for brevity on stack overflow
    state = request.GET.get('state',None)

    flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file(
        JSON_PATH,
        scopes=API_SCOPE,
        state=state)
    flow.redirect_uri = REDIRECT_URI

    #get the full URL that we are on, including all the "?param1=token&param2=key" parameters that google has sent us.
    authorization_response = request.build_absolute_uri()        

    #now turn those parameters into a token.
    flow.fetch_token(authorization_response=authorization_response)

    credentials = flow.credentials

    #now  we build the API service object    
    service = build('gmail', 'v1', credentials=credentials)
    #ok. awesome!
    #what email did they use? (this is just an example of how to use the api - you can skip this part if you want)
    profile = service.users().getProfile(userId="me").execute()
    email_address = profile['emailAddress']
    #ok. now we get the active manager
    manager = get_active_manager(request.user)
    #<lots of project specific code removed>
    #NOTE: 'manager' object is a project-specific type of object.
    #I store the auth token in it.
    #alright, if we get to here we have a valid manager object.

    #now lets create/update the credentials object in the DB.
    temp = save_credentials(manager,credentials)
    #now send them on their merry way that you've got access
    return HttpResponse("http://www.example.com")

here is the save/load functions i'm using. note that the 'manager' and 'Gmail_Connection_Token' objects are project specific objects that I'm saving tokens in.

def save_credentials(manager,credentials,valid=True):
    #this is the function that should be called to save the various tokens.
    #credentials is a google.oauth2.credentials.Credentials() object.
    #this saves it in a format that is easy to turn back 
    #into the same type of object in load_credentials(manager).
    #valid is, for the most part, always going to be true, but if for some reason its not
    #make sure to set that flag.
    #this returns the credentials as a dict (ignores the valid flag)
    #---------------------------------------
    #first we get or create the correct DB object
    try:
        creds = Gmail_Connection_Token.objects.get(manager=manager)
    except Gmail_Connection_Token.DoesNotExist:
        creds = Gmail_Connection_Token()
        creds.manager = manager
    #now we turn the passed in credentials obj into a dicts obj
    #note the expiry formatting
    temp = {
        'token': credentials.token,
        'refresh_token': credentials.refresh_token,
        'id_token':credentials.id_token,
        'token_uri': credentials.token_uri,
        'client_id': credentials.client_id,
        'client_secret': credentials.client_secret,
        'scopes': credentials.scopes,
        'expiry':datetime.datetime.strftime(credentials.expiry,'%Y-%m-%d %H:%M:%S')
    }
    #now we save it as a json_string into the creds DB obj
    creds.json_string = json.dumps(temp)
    #update the valid flag.
    creds.valid = valid
    #and save everythign in the DB
    creds.save()
    #and finally, return the dict we just created.
    return temp

and heres how I load the token whenever I need it:

def load_credentials(manager,ignore_valid=False):
    #this is the function that should be called to load a credentials object     from the database.
    #it loads, refreshes, and returns a     google.oauth2.credentials.Credentials() object.
    #raises a value error if valid = False 
    #------
    #NOTE: if 'ignore_valid' is True:
    #will NOT raise a value error if valid == False
    #returns a Tuple formated as (Credentails(),valid_boolean)
    #======================================
    try:
        creds = Gmail_Connection_Token.objects.get(manager=manager)
    except: 
        #if something goes wrong here, we want to just raise the error
        #and pass it to the calling function.
        raise #yes, this is proper syntax! (don't want to lose the stack     trace)
    #is it valid? do we raise an error?
    if not ignore_valid and not creds.valid:
        raise ValueError('Credentials are not valid.')
    #ok, if we get to here we load/create the Credentials obj()
    temp = json.loads(creds.json_string)
    credentials = google.oauth2.credentials.Credentials(
        temp['token'],
        refresh_token=temp['refresh_token'],
        id_token=temp['id_token'],
        token_uri=temp['token_uri'],
        client_id=temp['client_id'],
        client_secret=temp['client_secret'],
        scopes=temp['scopes'],
    )
    expiry = temp['expiry']
    expiry_datetime = datetime.datetime.strptime(expiry,'%Y-%m-%d %H:%M:%S')
    credentials.expiry = expiry_datetime
    #and now we refresh the token   
    #but not if we know that its not a valid token.
    if creds.valid:
        request = google.auth.transport.requests.Request()
        if credentials.expired:
            credentials.refresh(request)
    #and finally, we return this whole deal
    if ignore_valid:
        return (credentials,creds.valid)
    else:
        return credentials

thats pretty much everything. here is an example endpoint that shows how to use these functions whenever you need to access the Gmail api

@login_required
def test_endpoint(request):
    #get the project-specific manager database object we are using to store the tokens
    manager = get_active_manager(request.user)
    #and convert that manager object into the google credentials object
    credentials = load_credentials(manager)

    #do whatever you need the gmail api for here:
    msg = send_test_email(credentials)

    #when you're done, make sure to save/update the credentials in the DB for future use.
    save_credentials(manager,credentials)

    #then send your user on their merry way.
    return HttpResponse(msg)