Google OAuth2 not issuing a refresh token even with access_type='offline'?

Question

This question is similar to Not receiving Google OAuth refresh token, but I have already specified access_type='offline' as suggested in the comments to the accepted solution.

I'm writing a Django app to send calendar invites using the Google API which is basically an adaptation of the Flask example given at https://developers.google.com/api-client-library/python/auth/web-app, in which I've created a model GoogleCredentials to store credentials persistently in a database instead of in the session.

Here are the views:

import logging
from django.conf import settings
from django.shortcuts import redirect
from django.http import JsonResponse
from django.urls import reverse
from django.contrib.auth.decorators import login_required
import google.oauth2.credentials
import google_auth_oauthlib.flow
import googleapiclient.discovery
from lucy_web.models import GoogleCredentials

logger = logging.getLogger(__name__)


# Client configuration for an OAuth 2.0 web server application
# (cf. https://developers.google.com/identity/protocols/OAuth2WebServer)
# This is constructed from environment variables rather than from a
# client_secret.json file, since the Aptible deployment process would
# require us to check that into version control, which is not in accordance
# with the 12-factor principles.
# The client_secret.json containing this information can be downloaded from
# https://console.cloud.google.com/apis/credentials?organizationId=22827866999&project=cleo-212520
CLIENT_CONFIG = {'web': {
    'client_id': settings.GOOGLE_CLIENT_ID,
    'project_id': settings.GOOGLE_PROJECT_ID,
    'auth_uri': 'https://accounts.google.com/o/oauth2/auth',
    'token_uri': 'https://www.googleapis.com/oauth2/v3/token',
    'auth_provider_x509_cert_url': 'https://www.googleapis.com/oauth2/v1/certs',
    'client_secret': settings.GOOGLE_CLIENT_SECRET,
    'redirect_uris': settings.GOOGLE_REDIRECT_URIS,
    'javascript_origins': settings.GOOGLE_JAVASCRIPT_ORIGINS}}

# This scope will allow the application to manage the user's calendars
SCOPES = ['https://www.googleapis.com/auth/calendar']
API_SERVICE_NAME = 'calendar'
API_VERSION = 'v3'


@login_required
def authorize(request):
    authorization_url, state = _get_authorization_url(request)
    request.session['state'] = state
    return redirect(to=authorization_url)


@login_required
def oauth2callback(request):
    flow = _get_flow(request, state=request.session['state'])

    # Note: to test this locally, set OAUTHLIB_INSECURE_TRANSPORT=1 in your .env file
    # (cf. https://stackoverflow.com/questions/27785375/testing-flask-oauthlib-locally-without-https)
    flow.fetch_token(authorization_response=request.get_raw_uri())
    _save_credentials(user=request.user, credentials=flow.credentials)
    return redirect(to=reverse('create-meeting'))


@login_required
def create_meeting(request):
    # Retrieve the user's credentials from the database, redirecting
    # to the authorization page if none are found
    credentials = _get_credentials(user=request.user)
    if not credentials:
        return redirect(to=reverse('authorize'))

    calendar = googleapiclient.discovery.build(
        API_SERVICE_NAME, API_VERSION, credentials=credentials)

    calendars = calendar.calendarList().list().execute()

    return JsonResponse(calendars)


def _get_credentials(user):
    """
    Retrieve a user's google.oauth2.credentials.Credentials from the database.
    """
    try:
        _credentials = GoogleCredentials.objects.get(user=user)
    except GoogleCredentials.DoesNotExist:
        return

    return google.oauth2.credentials.Credentials(**_credentials.to_dict())


def _save_credentials(user, credentials):
    """
    Store a user's google.oauth2.credentials.Credentials in the database.
    """
    gc, _ = GoogleCredentials.objects.get_or_create(user=user)
    gc.update_from_credentials(credentials)


def _get_authorization_url(request):
    flow = _get_flow(request)

    # Generate URL for request to Google's OAuth 2.0 server
    return flow.authorization_url(
        # Enable offline access so that you can refresh an access token without
        # re-prompting the user for permission. Recommended for web server apps.
        access_type='offline',
        login_hint=settings.SCHEDULING_EMAIL,
        # Enable incremental authorization. Recommended as a best practice.
        include_granted_scopes='true')


def _get_flow(request, **kwargs):
    # Use the information in the client_secret.json to identify
    # the application requesting authorization.
    flow = google_auth_oauthlib.flow.Flow.from_client_config(
        client_config=CLIENT_CONFIG,
        scopes=SCOPES,
        **kwargs)

    # Indicate where the API server will redirect the user after the user completes
    # the authorization flow. The redirect URI is required.
    flow.redirect_uri = request.build_absolute_uri(reverse('oauth2callback'))
    return flow

Note that I have passed access_type='offline' to the flow.authorization_url(). Here is the GoogleCredentials model:

from django.db import models
from django.contrib.postgres.fields import ArrayField
from .timestamped_model import TimeStampedModel
from .user import User


class GoogleCredentials(TimeStampedModel):
    """
    Model for saving Google credentials to a persistent database (cf. https://developers.google.com/api-client-library/python/auth/web-app)
    The user's ID is used as the primary key, following https://github.com/google/google-api-python-client/blob/master/samples/django_sample/plus/models.py.
    (Note that we don't use oauth2client's CredentialsField as that library is deprecated).
    """
    user = models.OneToOneField(
        User,
        primary_key=True,
        limit_choices_to={'is_staff': True},
        # Deleting a user will automatically delete his/her Google credentials
        on_delete=models.CASCADE)
    token = models.CharField(max_length=255, null=True)
    refresh_token = models.CharField(max_length=255, null=True)
    token_uri = models.CharField(max_length=255, null=True)
    client_id = models.CharField(max_length=255, null=True)
    client_secret = models.CharField(max_length=255, null=True)
    scopes = ArrayField(models.CharField(max_length=255), null=True)

    def to_dict(self):
        """
        Return a dictionary of the fields required to construct
        a google.oauth2.credentials.Credentials object
        """
        return dict(
            token=self.token,
            refresh_token=self.refresh_token,
            token_uri=self.token_uri,
            client_id=self.client_id,
            client_secret=self.client_secret,
            scopes=self.scopes)

    def update_from_credentials(self, credentials):
        self.token = credentials.token
        self.refresh_token = credentials.refresh_token
        self.token_uri = credentials.token_uri
        self.client_id = credentials.client_id
        self.client_secret = credentials.client_secret
        self.scopes = credentials.scopes
        self.save()

With the development server running, if I go to localhost:8000/authorize (which is hooked up to the authorize() view) and I afterwards check the first credential, I see that the refresh_token is None:

(lucy-web-CVxkrCFK) bash-3.2$ python manage.py shell
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from lucy_web.models import *

In [2]: GoogleCredentials.objects.all()
Out[2]: <QuerySet [<GoogleCredentials: GoogleCredentials object (2154)>]>

In [3]: gc = GoogleCredentials.objects.first()

In [4]: gc.__dict__
Out[4]: 
{'_state': <django.db.models.base.ModelState at 0x111f91630>,
 'created_at': datetime.datetime(2018, 8, 15, 17, 58, 33, 626971, tzinfo=<UTC>),
 'updated_at': datetime.datetime(2018, 8, 15, 23, 8, 38, 634449, tzinfo=<UTC>),
 'user_id': 2154,
 'token': 'ya29foobar6tA',
 'refresh_token': None,
 'token_uri': 'https://www.googleapis.com/oauth2/v3/token',
 'client_id': '8214foobar13-unernto9l5ievs2pi0l6fir12fus1o46.apps.googleusercontent.com',
 'client_secret': 'bZt6foobarQj10y',
 'scopes': ['https://www.googleapis.com/auth/calendar']}

Initially, this is not a problem, but after a while, if I go to the create_meeting() view, I get a RefreshError which I've traced to this bit of source code in google.oauth2.credentials:

@_helpers.copy_docstring(credentials.Credentials)
def refresh(self, request):
    if (self._refresh_token is None or
            self._token_uri is None or
            self._client_id is None or
            self._client_secret is None):
        raise exceptions.RefreshError(
            'The credentials do not contain the necessary fields need to '
            'refresh the access token. You must specify refresh_token, '
            'token_uri, client_id, and client_secret.')

In other words, I need a refresh_token to prevent this error. Why is the Google API not returning one in this case?

Answer 1

Following the accepted answer more carefully, I found that I could get the refresh token by removing the web app's access to my account and adding it again. I navigated to https://myaccount.google.com/permissions and removed the access for the 'Cleo' app:

Then I went to localhost:8000/authorize (which is linked to the authorize() view) and looked up the saved credentials again, and they have a refresh token:

In [24]: from lucy_web.models import *

In [25]: gc = GoogleCredentials.objects.first()

In [26]: gc.__dict__
Out[26]: 
{'_state': <django.db.models.base.ModelState at 0x109133e10>,
 'created_at': datetime.datetime(2018, 8, 15, 17, 58, 33, 626971, tzinfo=<UTC>),
 'updated_at': datetime.datetime(2018, 8, 16, 22, 37, 48, 108105, tzinfo=<UTC>),
 'user_id': 2154,
 'token': 'ya29.Glv6BbcPkVoFfoobarHGifJUlEKP7kvwO5G1myTDOw9UYfl1LKAGxt',
 'refresh_token': '1/iafoobar4z1OxFtNljiLrmS0',
 'token_uri': 'https://www.googleapis.com/oauth2/v3/token',
 'client_id': '821409068013-unerntfoobarir12fus1o46.apps.googleusercontent.com',
 'client_secret': 'bZt6lfoobarpI8Qj10y',
 'scopes': ['https://www.googleapis.com/auth/calendar']}

Answer 2

Access is revoked after 1 hour, the way around that is by having a refresh_token. But this token is only given on the first authorization request (when the UI to allow calendar appears). To fix the issue, after not saving the token:

1. Revoke access to the app in: https://myaccount.google.com/u/0/permissions
2. Ask for access again
3. Save the refresh_token
4. Use the refresh_token

In normal operation, just:

1. Ask for access.
2. Save the refresh_token
3. Use the refresh_token