How to use Twisted to check Gmail with OAuth2.0 authentication

Question

I had a working IMAP client for Google mail, however it recently stopped working. I believe the problem is that gmail no longer allows TTL username/password logins, but now requires OAuth2.0.

I would like to know the best way to alter my example below such that my twisted IMAP client authenticates using OAuth2.0. (And doing so without Google API packages, if that's possible.)

Example using username/password login (no longer works)

class AriSBDGmailImap4Client(imap4.IMAP4Client):
    '''
    client to fetch and process SBD emails from gmail. the messages
    contained in the emails are sent to the AriSBDStationProtocol for
    this sbd modem.
    '''

    def __init__(self, contextFactory=None):
        imap4.IMAP4Client.__init__(self, contextFactory)

    @defer.inlineCallbacks
    def serverGreeting(self, caps):
        # log in
        try:
            # the line below no longer works for gmail
            yield self.login(mailuser, mailpass)
            try:
                yield self.uponAuthentication()
            except Exception as e:
                uponFail(e, "uponAuthentication")
        except Exception as e:
            uponFail(e, "logging in")

        # done. log out
        try:
            yield self.logout()
        except Exception as e:
            uponFail(e, "logging out")

    @defer.inlineCallbacks
    def uponAuthentication(self):
        try:
            yield self.select('Inbox')
            try:
                # read messages, etc, etc
                pass
            except Exception as e:
                uponFail(e, "searching unread")
        except Exception as e:
            uponFail(e, "selecting inbox")

I have a trivial factory for this client. It gets started by using reactor.connectSSL with Google mail's host url and port.

I have followed the directions at https://developers.google.com/gmail/api/quickstart/quickstart-python for an "installed app" (but I don't know if this was the right choice). I can run their "quickstart.py" example successfully.

My quick and dirty attempt (does not work)

    @defer.inlineCallbacks
    def serverGreeting(self, caps):
        # log in
        try:
            #yield self.login(mailuser, mailpass)
            flow = yield threads.deferToThread(
                oauth2client.client.flow_from_clientsecrets,
                filename=CLIENT_SECRET_FILE, 
                scope=OAUTH_SCOPE)
            http = httplib2.Http()
            credentials = yield threads.deferToThread( STORAGE.get )
            if credentials is None or credentials.invalid:
                parser = argparse.ArgumentParser(
                    parents=[oauth2client.tools.argparser])
                flags = yield threads.deferToThread( parser.parse_args )
                credentials = yield threads.deferToThread(
                    oauth2client.tools.run_flow,
                    flow=flow, 
                    storage=STORAGE,
                    flags=flags, http=http)
            http = yield threads.deferToThread(
                credentials.authorize, http)

            gmail_service = yield threads.deferToThread(
                apiclient.discovery.build,
                serviceName='gmail', 
                version='v1',
                http=http)

            self.state = 'auth'

            try:
                yield self.uponAuthentication()
            except Exception as e:
                uponFail(e, "uponAuthentication")
        except Exception as e:
            uponFail(e, "logging in")

        # done. log out
        try:
            yield self.logout()
        except Exception as e:
            uponFail(e, "logging out")

I basically just copied over "quickstart.py" into serverGreeting and then tried to set the client state to "auth".

This authenticates just fine, but then twisted is unable to select the inbox:

[AriSBDGmailImap4Client (TLSMemoryBIOProtocol),client] FAIL: Unknown command {random gibberish}

The random gibberish has letters and numbers and is different each time the select inbox command fails.

Thanks for your help!

Answer 1

After a lot of reading and testing, I was finally able to implement a working log-in to gmail using OAuth2.

One important note was that the 2-step process using a "service account" did NOT work for me. I'm still not clear why this process can't be used, but the service account does not seem to have access to the gmail in the same account. This is true even when the service account has "can edit" permissions and the Gmail API is enabled.

Useful References

Overview of using OAuth2 https://developers.google.com/identity/protocols/OAuth2

Guide for using OAuth2 with "Installed Applications" https://developers.google.com/identity/protocols/OAuth2InstalledApp

Guide for setting up the account to use OAuth2 with "Installed Applications" https://developers.google.com/api-client-library/python/auth/installed-app

Collection of OAuth2 routines without the full Google API https://code.google.com/p/google-mail-oauth2-tools/wiki/OAuth2DotPyRunThrough

Step 1 - Get a Google Client ID

Log in with the gmail account to https://console.developers.google.com/

Start a project, enable Gmail API and create a new client id for an installed application. Instructions at https://developers.google.com/api-client-library/python/auth/installed-app#creatingcred

Click the "Download JSON" button and save this file somewhere that will be inaccessible to the public (so probably not in the code repository).

Step 2 - Get the Google OAuth2 Python tools

Download the oauth2.py script from https://code.google.com/p/google-mail-oauth2-tools/wiki/OAuth2DotPyRunThrough

Step 3 - Get the authorization URL

Use the script from Step 2 to obtain a URL allowing you to authorize your Google project.

In the terminal:

python oauth2.py --user={[email protected]} --client_id={your client_id from the json file} --client_secret={your client_secret from the json file} --generate_oauth2_token

Step 4 -- Get the Authorization Code

Paste the URL from Step 3 into your browser and click the "accept" button.

Copy the code from the web page.

Paste the code into the terminal and hit enter. You will obtain:

 To authorize token, visit this url and follow the directions:   https://accounts.google.com/o/oauth2/auth?client_id{...}
 Enter verification code: {...}
 Refresh Token: {...}
 Access Token: {...}
 Access Token Expiration Seconds: 3600

Step 5 - Save the Refresh Token

Copy the refresh token from the terminal and save it somewhere. In this example, I save it to a json-formatted text file with the key "Refresh Token". But it could also be saved to a private database.

Make sure the refresh token cannot be accessed by the public!

Step 6 - Make a Twisted Authenticator

Here is a working example of an OAuth2 authenticator. It requires the oauth2.py script from Step 2.

import json
import oauth2

from zope.interface import implementer
from twisted.internet import threads

MY_GMAIL = {your gmail address}
REFRESH_TOKEN_SECRET_FILE = {name of your refresh token file from Step 5}
CLIENT_SECRET_FILE = {name of your cliend json file from Step 1}

@implementer(imap4.IClientAuthentication)
class GmailOAuthAuthenticator():
    authName     = "XOAUTH2"
    tokenTimeout = 3300      # 5 mins short of the real timeout (1 hour)

    def __init__(self, reactr):
        self.token   = None
        self.reactor = reactr
        self.expire  = None

    @defer.inlineCallbacks
    def getToken(self):

        if ( (self.token==None) or (self.reactor.seconds() > self.expire) ):
            rt = None
            with open(REFRESH_TOKEN_SECRET_FILE) as f:
                rt = json.load(f)

            cl = None
            with open(CLIENT_SECRET_FILE) as f:
                cl = json.load(f)

            self.token = yield threads.deferToThread(
                oauth2.RefreshToken,
                client_id = cl['installed']['client_id'], 
                client_secret = cl['installed']['client_secret'],
                refresh_token = rt['Refresh Token'] )

            self.expire = self.reactor.seconds() + self.tokenTimeout


    def getName(self):
        return self.authName

    def challengeResponse(self, secret, chal):
        # we MUST already have the token
        # (allow an exception to be thrown if not)

        t = self.token['access_token']

        ret = oauth2.GenerateOAuth2String(MY_GMAIL, t, False)

        return ret

Step 7 - Register the Authenitcator for the Protocol

In the IMAP4ClientFactory:

    def buildProtocol(self, addr):
        p = self.protocol(self.ctx)
        p.factory = self
        x = GmailOAuthAuthenticator(self.reactor)
        p.registerAuthenticator(x)
        return p

Step 8 - Use the Access Token to authenticate

Instead of using "login", get the Access Token (if necessary) and then use authenticate.

Altering the example code from the question:

    @defer.inlineCallbacks
    def serverGreeting(self, caps):
        # log in
        try:
            # the line below no longer works for gmail
            # yield self.login(mailuser, mailpass)
            if GmailOAuthAuthenticator.authName in self.authenticators:
                yield self.authenticators[AriGmailOAuthAuthenticator.authName].getToken()

            yield self.authenticate("")

            try:
                yield self.uponAuthentication()
            except Exception as e:
                uponFail(e, "uponAuthentication")
        except Exception as e:
            uponFail(e, "logging in")